binary/name gets confused under upgrades of osapi_compute and metadata when using wsgi files

What sequence of events got you here? Is this changing the deployment type post upgrade as well? It would be good to fully qualify what failed here, and why we didn't catch it, and if we need to put something into nova-manage to help.

Patch in progress - https://review.openstack.org/#/c/501359/

We're running nova-os-compute-api (nova/api/openstack/compute/wsgi.py), nova-metadata-api (nova/api/metadata/wsgi.py) and nova-placement-api (nova/api/openstack/placement/wsgi.py) under uwsgi. Each in separate containers, under venvs.

If you look at the code in the patch, or rather the code before the patch, it tries to update one service-binary 'osapi_compute'. If it doesn't find it, it tries to insert a *different* service-binary 'nova-osapi_compute'.

Reviewed: https://review.openstack.org/501359
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0b4a021e4224981e83dca67e7519458f9939f3cd
Submitter: Zuul
Branch: master

commit 0b4a021e4224981e83dca67e7519458f9939f3cd
Author: Erik Berg <email address hidden>
Date: Wed Sep 6 18:38:29 2017 +0000

    Fix binary name

    Before an upgrade, we have these type of entries in the db.

    MariaDB [nova]> SELECT id, host, `binary`, deleted, version FROM services;
    +----+--------------+--------------------+---------+---------+
    | id | host | binary | deleted | version |
    +----+--------------+--------------------+---------+---------+
    | 5 | r1-n-os-api | nova-osapi_compute | 0 | 16 |
    | 21 | r1-n-m-api | nova-metadata | 0 | 16 |

    The wsgi files we run basically boil down to something like

      NAME=metadata
      return wsgi_app.init_application(NAME)

    In the wsgi_app.py we see this function

      service_ref = objects.Service.get_by_host_and_binary(ctxt, host, name)

    Which results in a really big query, which again comes down to

      SELECT host, `binary` FROM services
        WHERE host = 'r1-n-m-api' AND `binary` == 'metadata'

    No results. service_ref is set to None. Carry on.

      if service_ref:
        #Nope.
      else:
        try:
          ...
          service_obj.host = host
          service_obj.binary = 'nova-%s' % name
          service_obj.create()

    Which results in a INSERT statement something like this;

      INSERT INTO services(host, `binary`, report_count, disabled, deleted, version)
        VALUES ('r1-n-m-api', 'nova-metadata', 0, 0, 0, 22)

      ERROR 1062 (23000): Duplicate entry 'r1-n-m-api-nova-metadata-0' for key 'uniq_services0host0binary0deleted'

    So the first suggested fix is to prepend 'nova-' to the name, and make both
    queries ask for 'nova-metadata'. There's also a check that it doesn't start
    with 'nova-', incase someone decides to prepend 'nova-' to the NAME= in the
    wsgi-file. Which migth be a litte overkill, but just a safeguard none the less.

    Change-Id: I58cf9a0115a98c78e5d2fb57c41c13ba6fac0fad
    Closes-bug: 1715463

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/524531

Reviewed: https://review.openstack.org/524531
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2d79baf6a3ce4529ba61a5efded7f043d89825a2
Submitter: Zuul
Branch: stable/pike

commit 2d79baf6a3ce4529ba61a5efded7f043d89825a2
Author: Erik Berg <email address hidden>
Date: Wed Sep 6 18:38:29 2017 +0000

    Fix binary name

    Before an upgrade, we have these type of entries in the db.

    MariaDB [nova]> SELECT id, host, `binary`, deleted, version FROM services;
    +----+--------------+--------------------+---------+---------+
    | id | host | binary | deleted | version |
    +----+--------------+--------------------+---------+---------+
    | 5 | r1-n-os-api | nova-osapi_compute | 0 | 16 |
    | 21 | r1-n-m-api | nova-metadata | 0 | 16 |

    The wsgi files we run basically boil down to something like

      NAME=metadata
      return wsgi_app.init_application(NAME)

    In the wsgi_app.py we see this function

      service_ref = objects.Service.get_by_host_and_binary(ctxt, host, name)

    Which results in a really big query, which again comes down to

      SELECT host, `binary` FROM services
        WHERE host = 'r1-n-m-api' AND `binary` == 'metadata'

    No results. service_ref is set to None. Carry on.

      if service_ref:
        #Nope.
      else:
        try:
          ...
          service_obj.host = host
          service_obj.binary = 'nova-%s' % name
          service_obj.create()

    Which results in a INSERT statement something like this;

      INSERT INTO services(host, `binary`, report_count, disabled, deleted, version)
        VALUES ('r1-n-m-api', 'nova-metadata', 0, 0, 0, 22)

      ERROR 1062 (23000): Duplicate entry 'r1-n-m-api-nova-metadata-0' for key 'uniq_services0host0binary0deleted'

    So the first suggested fix is to prepend 'nova-' to the name, and make both
    queries ask for 'nova-metadata'. There's also a check that it doesn't start
    with 'nova-', incase someone decides to prepend 'nova-' to the NAME= in the
    wsgi-file. Which migth be a litte overkill, but just a safeguard none the less.

    Change-Id: I58cf9a0115a98c78e5d2fb57c41c13ba6fac0fad
    Closes-bug: 1715463
    (cherry picked from commit 0b4a021e4224981e83dca67e7519458f9939f3cd)

This issue was fixed in the openstack/nova 16.0.4 release.