ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list

oooh found it!

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 793, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 274, in handle
    ocfg = convert_to_v3_apt_format(ocfg)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 758, in convert_to_v3_apt_format
    cfg = convert_v2_to_v3_apt_format(cfg)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 724, in convert_v2_to_v3_apt_format
    oldkey))

when I set the "Proxy" setting to "Don't use a proxy" I still get both of the errors in the events log, but I am able to successfully install packages

http://paste.ubuntu.com/26099075/

Setting this to Incomplete for MAAS, since it's not clear from if bad input data from MAAS is causing this traceback in cloud-init, or if the bug is only in cloud-init.

@jamesbeedy, can you tell us about how you have configured/customized your apt sources in MAAS?

ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list

Thats the error.
MAAS or the user sent configuration that had
 apt_preserve_sources_list: True
and
 apt:
   preserve_sources_list: False

Those are the same setting in 'Old' and 'New' config format. Cloud-init can't determine which one you meant, so it raised exception and recorded failure.

The curtin config would be good as well as any user-data that was provided.

@mponillo I have the curtin daily added, but not enabled (see screenshot)

@smoser I hadn't made any customizations to user-data or curtin config ... just what maas gives the node when it deploys.

I can deploy a node and reproduce the error at anytime, please let me know what files you want off the machine if any and I'll boot one up and swipe them for you.

Thanks for the clarification.

@pmontillo do you want a separate bug for the proxy issue?

Hi, in order to further investigate this bug we'll need to get some information about the node.
Please collect and attach the storage configuration for the node. You can get this information with:
 * maas 1.9 via cli
   maas <session> node get-curtin-config <system-id>
   maas <session> maas set-config name=curtin_verbose value=true

 * maas 2.0 via cli
   maas <session> machine get-curtin-config <system-id>

* Web UI: On the node details page in the installation output section at the bottom of the page

Also, in the installed environment, could you please collect /var/lib/cloud/ and /var/log/cloud-init*
thanks.

On MAAS 2.3 I still see this issue:

MAAS version: 2.3.0 (6434-gd354690-0ubuntu1~16.04.1

If I block curtin ephemeral environment I can see the competing config provided to debconf_selections:

/curtin/configs/config-001.cfg
debconf_selections: {grub2: grub2 grub2/update_nvram boolean false, maas: 'cloud-init cloud-init/datasources multiselect
    MAAS

    cloud-init cloud-init/maas-metadata-url string http://10.5.0.5/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_token_key=W9pshubrZ8StJAw5ss&oauth_consumer_key=r29m8b2F38n5jCEYZX&oauth_token_secret=G7ecZSKG8wsMHckxHCy9aE9Xh76U8bV4

    cloud-init cloud-init/local-cloud-config string apt:\n preserve_sources_list:
    false\n primary:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\n proxy:
    http://10.5.0.5:8000/\n security:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\napt_preserve_sources_list:
    true\napt_proxy: http://10.5.0.5:8000/\nmanage_etc_hosts: false\nmanual_cache_clean:
    true\nreporting:\n maas: {consumer_key: <REDACTED>, endpoint: ''http://10.5.0.5/MAAS/metadata/status/hhess8'',\n token_key:
    <REDACTED>, token_secret: <REDACTED>,\n type:
    webhook}\nsystem_info:\n package_mirrors:\n - arches: [i386, amd64]\n failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n search:\n primary:
    [''http://archive.ubuntu.com/ubuntu'']\n security: [''http://archive.ubuntu.com/ubuntu'']\n -
    arches: [default]\n failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n search:\n primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

    '}

Here is the curtin-config provided by MAAS to the node which also shows the competing apt_preserve_sources-list values on lines 49 and 51 of the paste

https://pastebin.ubuntu.com/26355967/

I tested the following fix suggestion on MAAS 2.3 (xenial image deployment) and it seems to bahave well with cloud-init, and avoids providing duplicate/conflicting apt_preferences to cloud-init via curtin

diff --git a/src/maasserver/compose_preseed.py b/src/maasserver/compose_preseed.py
index e15b3b2..2a41fbc 100644
--- a/src/maasserver/compose_preseed.py
+++ b/src/maasserver/compose_preseed.py
@@ -230,10 +230,9 @@ def compose_cloud_init_preseed(
         # See bug 1087183 for details.
         "manage_etc_hosts": False,
         "apt_preserve_sources_list": True,
- # Prevent the node from requesting cloud-init data on every reboot.
- # This is done so a machine does not need to contact MAAS every time
- # it reboots.
- "manual_cache_clean": True,
+ # On ubuntu >= xenial (cloud-init 17.2) manual_cache_clean is no
+ # longer required. But, you might want to set it for < xenial
+ # "manual_cache_clean": True,
     }
     # This is used as preseed for a node that's been installed.
     # This will allow cloud-init to be configured with reporting for
@@ -248,11 +247,6 @@ def compose_cloud_init_preseed(
         node.get_boot_rack_controller(), default_region_ip=default_region_ip)
     if apt_proxy:
         config['apt_proxy'] = apt_proxy
- # Add APT configuration for new cloud-init (>= 0.7.7-17)
- config.update(
- get_archive_config(
- node=node, preserve_sources=False,
- default_region_ip=default_region_ip))

     local_config_yaml = yaml.safe_dump(config)
     # this is debconf escaping

the above patch also tries addressing lp:1742479

The reason for dropping the config.update(get_archive_config) is because maas is already telling cloud-init to apt_preserve_sources_list: True above in line 230-ish. We don't need to resend curtin's apt configuration to the cloud_init_preseed config because that introduces conflicting apt settings.

MAAS provides curtin itself with the following apt configuration, which shows up in /curtin/configs/config-002.cfg
apt:
  preserve_sources_list: false
  primary:
  - arches: [default]
    uri: http://archive.ubuntu.com/ubuntu
  proxy: http://10.5.0.5:8000/
  security:
  - arches: [default]
    uri: http://archive.ubuntu.com/ubuntu
~

(the cloud-init deb_selections is provded via /curtin/configs/config-001.cfg)

The issue is that the deb_selections for cloud-init concatonated both curtin's apt config and cloud-init's apt config which says to cloud-init preserve apt sources: True. (This is seen in the cloud-init's local-cloud-config setting).

To block maas from rebooting out of curtin's ephemeral environment to test this, add the following to /etc/maas/preseeds/curtin_userdata:

early_commands:
  disable_reboot: touch /run/block-curtin-poweroff /tmp/block-poweroff /tmp/block-reboot

You can ssh into the instance with the ip address MAAS auto-allocates to the node.

Marking invalid for cloud-init as cloud-init's Traceback and error is 'desired' as it informs the cloud-init user of invalid cloud-config provided to cloud-init.

After install MAAS vanilla without any additional configuration or any extra repository and
MAASBuild-in proxy the error persist.

In addition the fix suggested in #12 works for MAAS 2.3 but NOT for MAAS 2.4

Configuration:

- Maas 2.4.2

- cloud-init --version
/usr/bin/cloud-init 18.3-9-g2e62cb8a-0ubuntu1~16.04.2

- cloud-init.log
https://pastebin.ubuntu.com/p/5vC6Zp4kvc/

- cloud-init-output.log
https://pastebin.ubuntu.com/p/bPcYVhXsM8/

- curtin-config
https://pastebin.ubuntu.com/p/Ys5qXPcCRb/

Thanks.

Is there any temporal fix for this error? I'm able to deploy Ubuntu 18.04 LTS but I get both errors described above:

Node post-installation failure - 'cloudinit' running modules for config

Node post-installation failure - 'cloudinit' running config-apt-configure with frequency once-per-instance

As I'm installing OpenStack using conjure-up, the installation fails.

Eduardo,

If the installation fails, its definitely not because of the bug above otherwise all ubuntu installations would fail.

This to me seems like a different issue, because the above message is only surface *after* the machine has already been marked as "Deployed". MAAS marks the machines deployed when cloud-init accesses the metadata *not* after cloud-init has run the metadata. This message is sent while cloud-init has bene running the metadata.

I would suggest you file a new bug report and include relevant information, including:

1. /var/log/maas/*.log
2. /var/log/maas/rsyslog/<machine-name>/<date>/messages
3. /var/log/cloud-init* from the deployed system
4. confirm if you are using Ubuntu mirrors or custom mirrors.

@Scott,

So it seems that curtin is always writing apt_preserve_sources_list: True. To test this, I'm ensuring that MAAS never sends apt_preserve_source_list (and this is reflected in the PR attached to this branch).

With those changes, the resulting curtin config is:

http://paste.ubuntu.com/p/s8GsHRQMhX/

As you can see, we never set apt_preserve_source_list. Hoewver, the deployed system results with:

ubuntu@guided-osprey:~$ cat /etc/cloud/cloud.cfg.d/curtin-preserve-sources.cfg
{apt_preserve_sources_list: true}

So it seems it is curtin that's adding an old cloud-init config style.

I spoke in IRC to Andres in response to his comment #19. curtin is
correct in telling cloud-init in the installed system to not re-render
/etc/apt/sources.list. However, it would be better if curtin did so using
the newer format where possible.

So, we will update curtin to use the following when cloud-init in the target allows:

apt:
  preserve_sources_list: true

Also, I have written a 'hackmd' doc at
https://hackmd.io/Gx5mVd_nRIixmAOkA2XMAA
that provides information on hte preserve_sources_list setting that
I will also attach here for posterity.

Please read it as I believe it has a complete discussion of this
setting wrt maas's use.

This bug is fixed with commit 40e6cc35 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=40e6cc35

This bug is believed to be fixed in curtin in version 18.2. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3075