nova-manage cell_v2 discover_hosts fails for IPv6 - ValueError: invalid literal for int() with base 10 - db connection URI gets its brackets eaten

The given traceback points out an oslo.db issue with parsing IPv6-based connection URI it seems?

FTR, looking at a random IPv6 job in CI [1], I can see that nova is also configured to consume IPv6 within square brackets:

#
# From nova.conf
#

# The SQLAlchemy connection string to use to connect to the database. (string
# value)
#connection=mysql://nova:nova@localhost/nova
connection=mysql+pymysql://nova_api:Ts8xQ2vqMydUprhl8rW4tSgf4@[fd00:fd00:fd00:2000::1ee]/nova_api?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf

[1] http://logs.rdoproject.org/02/661702/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/65f1ab5/

something is eating the brackets before it gets to SQLAlchemy:

>>> from sqlalchemy.engine import url

# with brackets, happy
>>> url.make_url("mysql+pymysql://nova_api:LP1VoffTn36gPLU8tz67YYEV8@[fd12:3456:789a:1::3]/nova_api?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf")
mysql+pymysql://nova_api:***@[fd12:3456:789a:1::3]/nova_api?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo

# take the two brackets out, identical error

>>> url.make_url("mysql+pymysql://nova_api:LP1VoffTn36gPLU8tz67YYEV8@fd12:3456:789a:1::3/nova_api?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/classic/dev/sqlalchemy/lib/sqlalchemy/engine/url.py", line 229, in make_url
    return _parse_rfc1738_args(name_or_url)
  File "/home/classic/dev/sqlalchemy/lib/sqlalchemy/engine/url.py", line 288, in _parse_rfc1738_args
    return URL(name, **components)
  File "/home/classic/dev/sqlalchemy/lib/sqlalchemy/engine/url.py", line 71, in __init__
    self.port = int(port)
ValueError: invalid literal for int() with base 10: '3456:789a:1::3'

if nova.conf has the correct ipv6 URL inside it and then it's getting munged when nova consumes it would this be potentially nova as well ?

I think we'd need some better background on how to reproduce this and also why isn't it happening on current CI.

So, I have had this reproduced for a while, just had'nt time to dig deeper.

This happens when running 'nova_cell_v2_discover_hosts':

    "Error running ['podman', 'run', '--name', 'nova_cell_v2_discover_hosts', '--label', 'config_id=tripleo_step5', '--label', 'container_name=nova_cell_v2_discover_hosts', '--label', 'managed_by=paunch', '--lab
el', 'config_data={\"start_order\": 0, \"command\": \"/usr/bin/bootstrap_host_exec nova_ironic su nova -s /bin/bash -c \\'/container-config-scripts/pyshim.sh /container-config-scripts/nova_cell_v2_discover_hosts
.py\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:
/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/t
ls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/nova/etc/my.cnf.d/:/etc/my.cnf.d/:r
o\", \"/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/container-config-scripts/:/container-config-scripts/\"], \"image\": \"docker.io/tripleomaster/ce
ntos-binary-nova-compute-ironic:current-tripleo\", \"detach\": false, \"net\": \"host\"}', '--conmon-pidfile=/var/run/nova_cell_v2_discover_hosts.pid', '--log-driver', 'json-file', '--log-opt', 'path=/var/log/co
ntainers/stdouts/nova_cell_v2_discover_hosts.log', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca
-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/cert
s/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_ho
sts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/nova/etc/my.cnf.d/:/etc/my.cnf.d/:ro', '--volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/n
ova:/var/log/nova', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/', 'docker.io/tripleomaster/centos-binary-nova-compute-ironic:current-tripleo', '/usr/bin/bootstrap_host_exec', 'nova_ir
onic', 'su', 'nova', '-s', '/bin/bash', '-c', \"'/container-config-scripts/pyshim.sh\", \"/container-config-scripts/nova_cell_v2_discover_hosts.py'\"]. [1]",

-- The scipts simply call's nova-manage afict.

import logging
import os
import random
import subprocess
import sys
import time

random.seed()

debug = os.getenv('__OS_DEBUG', 'false')

if debug.lower() == 'true':
    loglevel = logging.DEBUG
else:
    loglevel = logging.INFO

logging.basicConfig(stream=sys.stdout, level=loglevel)
LOG = logging.getLogger('no...

So, I got into the container and tried running the nova-manage command manually.
The same error happens.

afaict the config is good:
bash-4.2$ grep '3456:789a:1::3' /etc/nova/nova.conf
connection=mysql+pymysql://nova_api:LP1VoffTn36gPLU8tz67YYEV8@[fd12:3456:789a:1::3]/nova_api?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf
connection=mysql+pymysql://nova:LP1VoffTn36gPLU8tz67YYEV8@[fd12:3456:789a:1::3]/nova?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf
api_servers=http://[fd12:3456:789a:1::3]:9292
auth_url=http://[fd12:3456:789a:1::3]:5000
api_endpoint=http://[fd12:3456:789a:1::3]:6385/v1
www_authenticate_uri=http://[fd12:3456:789a:1::3]:5000
auth_uri=http://[fd12:3456:789a:1::3]:5000
auth_url=http://[fd12:3456:789a:1::3]:35357
url=http://[fd12:3456:789a:1::3]:9696
auth_url=http://[fd12:3456:789a:1::3]:5000/v3
auth_url=http://[fd12:3456:789a:1::3]:5000

bash-4.2$ /usr/bin/nova-manage cell_v2 discover_hosts --by-service --verbose
Found 2 cell mappings.
Skipping cell0 since it does not contain hosts.
Getting computes from cell 'default': 4165e8f0-6bbc-4b61-86c7-208ba08f0589
An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 2212, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1303, in discover_hosts
    by_service)
  File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 265, in discover_hosts
    by_service)
  File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 221, in _check_and_create_host_mappings
    ctxt, 'nova-compute', include_disabled=True)
  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
    result = fn(cls, context, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 591, in get_by_binary
    context, binary, include_disabled=include_disabled)
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 145, in service_get_all_by_binary
    include_disabled=include_disabled)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 228, in wrapped
    with ctxt_mgr.reader.using(context):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
    context=context) as resource:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session
    bind=self.connection, mode=self.mode)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 418, in _create_session
    self._start()
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 510, in _start
    engine_args, maker_args)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 534, in _setup_for_connection
    sql_connection=sql_connection, **engine_kwargs)
  File "/usr/lib/python2.7/site-packages/debtc...

Added debug in: oslo_db/sqlalchemy/enginefacade.py

 455 def _url_args_for_conf(self, conf):
 456 retval = self._args_for_conf(self._url_cfg, conf)
 457 print('DEBUG _url_args_for_conf: %s' % retval)
 458 return retval

 483 def _start(self, conf=False, connection=None, slave_connection=None):
 484 print('DEBUG _start conf: %s' % conf)
 485 with self._start_lock:
 486 # self._started has been checked on the outside
 487 # when _start() was called. Within the lock,
 488 # check the flag once more to detect the case where
 489 # the start process proceeded while this thread was waiting
 490 # for the lock.
 491 if self._started:
 492 return
 493 if conf is False:
 494 conf = cfg.CONF
 495
 496 # perform register_opts() local to actually using
 497 # the cfg.CONF to maintain exact compatibility with
 498 # the EngineFacade design. This can be changed if needed.
 499 if conf is not None:
 500 conf.register_opts(options.database_opts, 'database')
 501
 502 url_args = self._url_args_for_conf(conf)

DEBUG _start conf: False
DEBUG _url_args_for_conf: {'connection': 'mysql+pymysql://nova_api:LP1VoffTn36gPLU8tz67YYEV8@[fd12:3456:789a:1::3]/nova_api?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf', 'slave_connection': None}

Found 2 cell mappings.
Skipping cell0 since it does not contain hosts.
Getting computes from cell 'default': 4165e8f0-6bbc-4b61-86c7-208ba08f0589

DEBUG _start conf: False
DEBUG _url_args_for_conf: {'connection': u'mysql+pymysql://nova:LP1VoffTn36gPLU8tz67YYEV8@fd12:3456:789a:1::3/nova?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf', 'slave_connection': None}

An error has occurred:
Traceback (most recent call last):
 <--- snip --->

Something must be editing that config somewhere?

Fix proposed to branch: master
Review: https://review.opendev.org/664559

bash-4.2$ /usr/bin/nova-manage cell_v2 discover_hosts --by-service --verbose
Found 2 cell mappings.
Skipping cell0 since it does not contain hosts.
Getting computes from cell 'default': 7e326f37-322a-4cdc-94a2-d4c3db58ac4a
An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 2212, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1303, in discover_hosts
    by_service)
  File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 265, in discover_hosts
    by_service)
  File "/usr/lib/python2.7/site-packages/nova/objects/host_mapping.py", line 221, in _check_and_create_host_mappings
    ctxt, 'nova-compute', include_disabled=True)
  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
    result = fn(cls, context, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 591, in get_by_binary
    context, binary, include_disabled=include_disabled)
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 145, in service_get_all_by_binary
    include_disabled=include_disabled)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 228, in wrapped
    with ctxt_mgr.reader.using(context):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1064, in _transaction_scope
    context=context) as resource:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session
    bind=self.connection, mode=self.mode)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 418, in _create_session
    self._start()
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 510, in _start
    engine_args, maker_args)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 534, in _setup_for_connection
    sql_connection=sql_connection, **engine_kwargs)
  File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 153, in create_engine
    url = sqlalchemy.engine.url.make_url(sql_connection)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 225, in make_url
    return _parse_rfc1738_args(name_or_url)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 284, in _parse_rfc1738_args
    return URL(name, **components)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 71, in __init__
    self.port = int(port)
ValueError: invalid literal for int() with base 10: '3456:789a:1::3'

######################################################################
###### Apply patch: https://review.opendev.org/664559 manually. #####
bash-4.2$ vi lib/python2.7/site-packages/nova/objects/cell_mapping.py ...

Change abandoned by Harald Jensås (<email address hidden>) on branch: master
Review: https://review.opendev.org/664559
Reason: Thanks to Dan! After looking at this more I conclude that you are indeed correct.

If I got it right, the brackets must be added to the database-connection template when creating the cell. nova-manage cell_v2 create_cell --name=foo --database_connection=$TEMPLATE

Too me it seems very backward to have the need to use different template's there based on weather hostname is a fqdn, IPv4 address or IPv6 address. But it's also not right to change the current behavior.

Change abandoned by Harald Jensås (<email address hidden>) on branch: master
Review: https://review.opendev.org/664559

Fix proposed to branch: master
Review: https://review.opendev.org/666621

Reviewed: https://review.opendev.org/666621
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=5e66a91ef0ef33caeaec1efc39bdaaf660a1e8a4
Submitter: Zuul
Branch: master

commit 5e66a91ef0ef33caeaec1efc39bdaaf660a1e8a4
Author: Harald Jensås <email address hidden>
Date: Thu Jun 20 17:39:24 2019 +0200

    Set IPv6 params True if undercloud local_ip is IPv6

    When deploying the undercloud using an IPv6 control plane
    the set various parameters for IPv6 to True.

    Change-Id: I101c4c70f2804779089ba25720141a9320270edc
    Closes-Bug: #1831315

This issue was fixed in the openstack/python-tripleoclient 12.1.0 release.

From the abandoned nova review https://review.opendev.org/#/c/664559/ comment it seems that the problem is well understood and nova can actually be configured with ipv6 addresses. There are certain inconvenience with the current way of templateig so I Confirm this bug but set it as wishlist.