LXD Charm

Bug #1825393
Comment #5

Comment 5 for bug 1825393

Revision history for this message

Narinder Gupta (narindergupta) wrote on 2019-04-18: Re: [Bug 1825393] Re: upgrade charm from 19.01 to 19.04 fails on initalize storage pool

I agreed Alex and appropriate to have fix no 1.

1. Add an 'has_storage()' test and then see if the LXD_POOL storage is set
up; if it is then exit.

Thanks and Regards,
Narinder Gupta
Canonical, Ltd.
+1.281.736.5150

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Thu, Apr 18, 2019 at 2:45 PM Alex Kavanagh <email address hidden>
wrote:

> Thanks Narinda
>
> I think I've worked out what the bug is, although I'm not quite sure the
> best way to fix it.
>
> -- Notes for fix:
>
> The main issue is in hooks/lxd_utils.py at line 192:
>
> if filesystem_mounted('/var/lib/lxd'):
> log('/var/lib/lxd already configured, skipping')
> return
>
> With LXD 3.x.x and greater, lxc storage is used, and so there is no
> filesystem mounted at /var/lib/lxd. Therefore, whenever the config-
> changed hook fires, it will try to re-setup the block device as the
> charm hasn't detected that it's already set up.
>
> There are a couple of ways to fix the bug:
>
> 1. Add an 'has_storage()' test and then see if the LXD_POOL storage is set
> up; if it is then exit.
> 2. Use the the fact that the /var/lib/lxd directory exists as evidence
> that the storage device has been set-up; it's more efficient than 1. but
> maybe not as dependable (e.g. if the previous config-change failed due to
> the block device not being mounted, but the directory was made).
>
> I'm leaning towards 1. as an appropriate fix.
>
> ** Changed in: charm-lxd
> Status: New => Confirmed
>
> ** Changed in: charm-lxd
> Importance: Undecided => High
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1825393
>
> Title:
> upgrade charm from 19.01 to 19.04 fails on initalize storage pool
>
> Status in OpenStack LXD Charm:
> Confirmed
>
> Bug description:
> Deploy using 19.02 charm then upgrade to 19.04 charm. config-change
> hook error.
>
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd',
> 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
> unit-lxd-1: 16:21:16 ERROR juju.worker.uniter.operation hook
> "config-changed" failed: exit status 1
> unit-lxd-0: 16:21:14 INFO unit.lxd/0.juju-log Configuring LXD container
> storage
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | NAME |
> DESCRIPTION | DRIVER | SOURCE | USED BY |
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | lxd |
> | btrfs | 4e0bbaa6-6fad-4428-8ffd-81828cdf5746 | 0 |
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Error: The storage
> pool already exists
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Traceback (most
> recent call last):
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 173, in
> <module>
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed main()
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 166, in
> main
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> hooks.execute(sys.argv)
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/charmhelpers/core/hookenv.py",
> line 914, in execute
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> self._hooks[hook_name]()
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 86, in
> config_changed
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> configure_lxd_block()
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/lxd_utils.py", line 233, in
> configure_lxd_block
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> config_btrfs(dev)
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/lxd_utils.py", line 246, in
> config_btrfs
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed check_call(cmd)
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed File
> "/usr/lib/python2.7/subprocess.py", line 190, in check_call
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed raise
> CalledProcessError(retcode, cmd)
> unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd',
> 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
> unit-lxd-0: 16:21:14 ERROR juju.worker.uniter.operation hook
> "config-changed" failed: exit status 1
> unit-lxd-1: 16:21:16 INFO unit.lxd/1.juju-log Configuring LXD container
> storage
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed | NAME |
> DESCRIPTION | DRIVER | SOURCE | USED BY |
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed | lxd |
> | btrfs | d33d8b1b-8096-48a4-8a4c-447168e6d049 | 0 |
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> +------+-------------+--------+--------------------------------------+---------+
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed Error: The storage
> pool already exists
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed Traceback (most
> recent call last):
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed File
> "/var/lib/juju/agents/unit-lxd-1/charm/hooks/config-changed", line 173, in
> <module>
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed main()
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed File
> "/var/lib/juju/agents/unit-lxd-1/charm/hooks/config-changed", line 166, in
> main
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> hooks.execute(sys.argv)
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed File
> "/var/lib/juju/agents/unit-lxd-1/charm/hooks/charmhelpers/core/hookenv.py",
> line 914, in execute
> unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> self._hooks[hook_name]()
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-lxd/+bug/1825393/+subscriptions
>

I agreed Alex and appropriate to have fix no 1.

1. Add an 'has_storage()' test and then see if the LXD_POOL storage is set
up; if it is then exit.

Thanks and Regards,
Narinder Gupta
Canonical, Ltd.
+1.281.736.5150

Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com

On Thu, Apr 18, 2019 at 2:45 PM Alex Kavanagh <1825393@bugs.launchpad.net>
wrote:

> Thanks Narinda
>
> I think I've worked out what the bug is, although I'm not quite sure the
> best way to fix it.
>
> -- Notes for fix:
>
> The main issue is in hooks/lxd_utils.py at line 192:
>
>     if filesystem_mounted('/var/lib/lxd'):
>         log('/var/lib/lxd already configured, skipping')
>         return
>
> With LXD 3.x.x and greater, lxc storage is used, and so there is no
> filesystem mounted at /var/lib/lxd.  Therefore, whenever the config-
> changed hook fires, it will try to re-setup the block device as the
> charm hasn't detected that it's already set up.
>
> There are a couple of ways to fix the bug:
>
> 1. Add an 'has_storage()' test and then see if the LXD_POOL storage is set
> up; if it is then exit.
> 2. Use the the fact that the /var/lib/lxd directory exists as evidence
> that the storage device has been set-up; it's more efficient than 1. but
> maybe not as dependable (e.g. if the previous config-change failed due to
> the block device not being mounted, but the directory was made).
>
> I'm leaning towards 1. as an appropriate fix.
>
> ** Changed in: charm-lxd
>        Status: New => Confirmed
>
> ** Changed in: charm-lxd
>    Importance: Undecided => High
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1825393
>
> Title:
>   upgrade charm from 19.01 to 19.04 fails on initalize storage pool
>
> Status in OpenStack LXD Charm:
>   Confirmed
>
> Bug description:
>   Deploy using 19.02 charm then upgrade to 19.04 charm. config-change
>   hook error.
>
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd',
> 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
>   unit-lxd-1: 16:21:16 ERROR juju.worker.uniter.operation hook
> "config-changed" failed: exit status 1
>   unit-lxd-0: 16:21:14 INFO unit.lxd/0.juju-log Configuring LXD container
> storage
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | NAME |
> DESCRIPTION | DRIVER |                SOURCE                | USED BY |
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed | lxd  |
>  | btrfs  | 4e0bbaa6-6fad-4428-8ffd-81828cdf5746 | 0       |
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> +------+-------------+--------+--------------------------------------+---------+
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Error: The storage
> pool already exists
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed Traceback (most
> recent call last):
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed   File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 173, in
> <module>
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed     main()
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed   File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 166, in
> main
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
>  hooks.execute(sys.argv)
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed   File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/charmhelpers/core/hookenv.py",
> line 914, in execute
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
>  self._hooks[hook_name]()
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed   File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/config-changed", line 86, in
> config_changed
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
>  configure_lxd_block()
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed   File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/lxd_utils.py", line 233, in
> configure_lxd_block
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
>  config_btrfs(dev)
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed   File
> "/var/lib/juju/agents/unit-lxd-0/charm/hooks/lxd_utils.py", line 246, in
> config_btrfs
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed     check_call(cmd)
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed   File
> "/usr/lib/python2.7/subprocess.py", line 190, in check_call
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed     raise
> CalledProcessError(retcode, cmd)
>   unit-lxd-0: 16:21:14 DEBUG unit.lxd/0.config-changed
> subprocess.CalledProcessError: Command '['lxc', 'storage', 'create', 'lxd',
> 'btrfs', 'source=/dev/sdh']' returned non-zero exit status 1
>   unit-lxd-0: 16:21:14 ERROR juju.worker.uniter.operation hook
> "config-changed" failed: exit status 1
>   unit-lxd-1: 16:21:16 INFO unit.lxd/1.juju-log Configuring LXD container
> storage
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> +------+-------------+--------+--------------------------------------+---------+
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed | NAME |
> DESCRIPTION | DRIVER |                SOURCE                | USED BY |
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> +------+-------------+--------+--------------------------------------+---------+
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed | lxd  |
>  | btrfs  | d33d8b1b-8096-48a4-8a4c-447168e6d049 | 0       |
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
> +------+-------------+--------+--------------------------------------+---------+
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed Error: The storage
> pool already exists
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed Traceback (most
> recent call last):
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed   File
> "/var/lib/juju/agents/unit-lxd-1/charm/hooks/config-changed", line 173, in
> <module>
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed     main()
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed   File
> "/var/lib/juju/agents/unit-lxd-1/charm/hooks/config-changed", line 166, in
> main
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
>  hooks.execute(sys.argv)
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed   File
> "/var/lib/juju/agents/unit-lxd-1/charm/hooks/charmhelpers/core/hookenv.py",
> line 914, in execute
>   unit-lxd-1: 16:21:16 DEBUG unit.lxd/1.config-changed
>  self._hooks[hook_name]()
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-lxd/+bug/1825393/+subscriptions
>