nova-cloud-controller db sync sometimes fails in HA mode

I think are hitting this bug too. Attached is my log.

I filed https://bugs.launchpad.net/charms/+source/nova-cloud-controller/+bug/1347245 with another backtrace I had, this might be a different bug.

The db-changed hook is gated to ensure that only one unit runs the db-sync task:

@hooks.hook('shared-db-relation-changed')
@restart_on_change(restart_map())
def db_changed():
    if 'shared-db' not in CONFIGS.complete_contexts():
        log('shared-db relation incomplete. Peer not ready?')
        return
    CONFIGS.write_all()

    if eligible_leader(CLUSTER_RES):
        migrate_database()
        log('Triggering remote cloud-compute restarts.')
        [compute_joined(rid=rid, remote_restart=True)
         for rid in relation_ids('cloud-compute')]

The 'eligible_leader' function does a few checks

1) if fully clustered, then the owner of the VIP is the leader
2) if not fully clustered, the oldest service unit in the 'cluster' relation is declared the leader

If in the event of 2) the cluster relation is not fully formed with all service units prior to the shared-db relation being made, I could see how 2 nova-cc units both think they are the leader and you hit a race; do you see the db sync being run on multiple service units?

On 23 July 2014 14:01, James Page <email address hidden> wrote:

> The 'eligible_leader' function does a few checks
>
> 1) if fully clustered, then the owner of the VIP is the leader
> 2) if not fully clustered, the oldest service unit in the 'cluster'
> relation is declared the leader
>
> If in the event of 2) the cluster relation is not fully formed with all
> service units prior to the shared-db relation being made, I could see
> how 2 nova-cc units both think they are the leader and you hit a race;
> do you see the db sync being run on multiple service units?
>

Yes.

Given it's not possible for a user of the Juju API to know when a relation
is "fully formed" I'm not sure if/how we can work around this.

We definitely hit this problem during original HA implementation and testing; two things where done to work-around this (pending any sort of leader election function from Juju itself):

1) juju deployer deploys service units and waits for them all to start before adding relations

2) peer relations should always fire before other relations when you introduce a new unit into an existing service

Are you waiting for all service units to fully start prior to adding relations between services?

I raised bug 1228243 a while back to request this feature.

Just hit this with current /next charms (rev 154) - http://paste.ubuntu.com/10827860/

Just hit this one more time at Cisco. Could attach a ton of logs if necessary, but it's the same failure mode.

2015-08-28 20:22:38 INFO shared-db-relation-changed 2015-08-28 20:22:38.017 56639 CRITICAL nova [-] OperationalError: (OperationalError) (1050, "Table 'instances' already exists") "\nCREATE TABLE instances (\n\tcrea
....

2015-08-28 20:22:38 INFO shared-db-relation-changed subprocess.check_output(cmd)
2015-08-28 20:22:38 INFO shared-db-relation-changed File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
2015-08-28 20:22:38 INFO shared-db-relation-changed raise CalledProcessError(retcode, cmd, output=output)
2015-08-28 20:22:38 INFO shared-db-relation-changed subprocess.CalledProcessError: Command '['nova-manage', 'db', 'sync']' returned non-zero exit status 1

        u"cs:trusty/nova-cloud-controller-60",
        config={"openstack-origin": "cloud:trusty-icehouse",

Our juju charm settings from the above failed deployment

I'm going to mark this as fix released since leader elections landed in nova-cloud-controller on June 9 2015 and that should fix the issue reported in this bug. If that's not the case please feel free to open the bug back up.

commit eaaaec38ddd6b49ce8be530529b6ffaf165ba6e1
Merge: b651ebf d4b768f
Author: James Page <email address hidden>
Date: Tue Jun 9 10:59:06 2015 +0100

    Add support for leader-election