puppet 5 package incompatible with ruby 3 / ubuntu 22.04

Status changed to 'Confirmed' because the bug affects multiple users.

At least as far as I have tested with my puppet installations, it is just a single line of code that needs to be fixed:

Line 124 in /usr/lib/ruby/vendor_ruby/puppet/file_system/file_impl.rb

    FileUtils.symlink(path, dest, options)

where ruby 3 accepts only two unnamed parameters here. Removing the third parameter options here make puppet 5 from the ubuntu package run for me, but I am not actually sure and haven't seen a case where the options are actually used. Technically, and since options is a hash of options, the natural way would be to use **options instead, but I am not sure, what kind of options could come and whether they conform to the ruby 3 API.

However, for the moment my workaround and quickfix is to just insert ** before options or to completely omit this third parameter.

This is not really a good solution, since puppet 5 is out of maintenance anyway and should be replaced by puppet 7, but maybe this could buy some time to allow creating a puppet 7 package or wait for the debian team in a relaxed way.

Thanks for the reply Hadmut. We should indeed try to fix the current version we have in Ubuntu while Debian maintainers sort out the update to a newer version.

What you described is indeed one of the issue we have with version 5.5.22-4, in Jammy and Kinetic. I did some investigation in the upstream bug tracker and I found some fixes that we might want to backport to this version:

* PUP-10537 (https://tickets.puppetlabs.com/browse/PUP-10537)

This is the upstream fix for the issue you mentioned. This is the commit: https://github.com/puppetlabs/puppet/commit/6af09225b3b962547a

* PUP-11045 (https://tickets.puppetlabs.com/browse/PUP-11045)

This fix is related to OpenSSL 3 which is the default since Jammy. This is the commit: https://github.com/puppetlabs/puppet/commit/c8ad6590c1860cdc

* PUP-11439 (https://tickets.puppetlabs.com/browse/PUP-11439)

This issue is also related to OpenSSL 3 but it does not have a fix commited. Checking the mentioned file I also found PUP-11082 which needs more careful consideration and it might be worth backporting it. This is the commit: https://github.com/puppetlabs/puppet/commit/78712feb5dd5456

* PUP-11048 (https://tickets.puppetlabs.com/browse/PUP-11048)

This is also related to ruby 3.0 transition. This is the commit: https://github.com/puppetlabs/puppet/commit/93c86d6dc8

After a more in-deep analysis and trying to backport the patches mentioned above I noticed some of them are not even applicable to the version we have in the archive. The following patches will be added to Kinectic, and later on backported to Jammy:

* PUP-10537 (https://tickets.puppetlabs.com/browse/PUP-10537)

This is the upstream fix for the issue you mentioned. This is the commit: https://github.com/puppetlabs/puppet/commit/6af09225b3b962547a

* PUP-11045 (https://tickets.puppetlabs.com/browse/PUP-11045)

This fix is related to OpenSSL 3 which is the default since Jammy. This is the commit: https://github.com/puppetlabs/puppet/commit/c8ad6590c1860cdc

Hello Hadmut, or anyone else affected,

Accepted puppet into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/puppet/5.5.22-4ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Can confirm the package solves this issue. When running puppet agent --test I now have other error messages like this:

Error: /File[/var/cache/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': undefined method `to_sym' for nil:NilClass
Error: /File[/var/cache/puppet/facts.d]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///pluginfacts

(and same with some modules I'm using from puppet forge)

but the agent finishes its run.

Thanks for testing the proposed package Adrián. Unfortunately, the output you presented does not help much in identifying the root cause of the error you got. Could you please try to pass '--verbose' and '--debug' to see if we get more useful information? Just to make sure, by the end of the command execution you got return code 0, right? Is the state of the system the same as expected by you? Or something is missing?

Hi Lucas! Here's the output of puppet agent --test --verbose --debug, near the error messages. puppet agent does exit with 0 and otherwise does the run properly (my test was to delete one of the symlinks I have in my node specification and it is correctly recreated). I'll try to build a new container to test this further, as I only have the one Jammy container and it was already fully configured (FWIW, when I only hand-applied the fix for the symlink function -- Puppet::FileSystem::symlink IIRC -- it didn't throw these errors).

Thanks for the follow-up!

I apologize for not responding earlier to testing the solutiion for the bug I've reported, but I'm currently on a trip, was busy the last days, and don't have my ttesting environment with me. I'll test as soon as poosible.

Hi! Just created a test VM with 22.04 (updated to latest), and I have such code:

        file { '/etc/localtime':
                ensure => 'link',
                target => $osfamily ? {
                        'Debian' => '/usr/share/zoneinfo/posix/America/Los_Angeles',
                        default => '/usr/share/zoneinfo/America/Los_Angeles',
                },

and initially I had this and only this error:

Error: wrong number of arguments (given 3, expected 2)
Error: /Stage[main]/Linux::Vm/File[/etc/localtime]/target: change from '/usr/share/zoneinfo/Etc/UTC' to '/usr/share/zoneinfo/posix/America/Los_Angeles' failed: wrong number of arguments (given 3, expected 2)

but now, with proposed version I'm getting something different:

Error: /File[/var/cache/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': undefined method `to_sym' for nil:NilClass
Error: /File[/var/cache/puppet/facts.d]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///pluginfacts
Error: /File[/var/cache/puppet/lib]: Failed to generate additional resources using 'eval_generate': undefined method `to_sym' for nil:NilClass
Error: /File[/var/cache/puppet/locales]: Failed to generate additional resources using 'eval_generate': undefined method `to_sym' for nil:NilClass

I enabled proposed repo and update puppet to that version:

puppet:
  Installed: 5.5.22-4ubuntu0.1
  Candidate: 5.5.22-4ubuntu0.1
  Version table:
 *** 5.5.22-4ubuntu0.1 400
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     5.5.22-4 500
        500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

and this one:
Error: /Stage[main]/Linux::Common/File[/etc/apt/apt.conf.d]: Failed to generate additional resources using 'eval_generate': undefined method `to_sym' for nil:NilClass

Hi,

sorry, I'm currently beeing on a trip, sitting in a hotel with rather limited technical equipment.

I also ran into this undefined method `to_sym' for nil:NilClass problem, but I also found the error message

Failed to load library 'msgpack' for feature 'msgpack': cannot load such file -- msgpack

After

apt install ruby-msgpack

that error was gone.

@Adrian: please check. If this helps for you too, then it is just a missing dependency.

Naw, sorry, didn't fix the to_sym message, just fixed the missing msgpack.

According to the Gemfile, msgpack is not a mandatory dependency, so in theory the lack of ruby-msgpack should not be a big deal. From Gemfile:

gem "msgpack", :require => false

I did some debugging, and the problem seems to appear in
/usr/lib/ruby/vendor_ruby/puppet/settings.rb

when interpolating $confdir in some internal config strings.

To be honest, the bug initially reported here is fixed, I just tried what is described in the Test Plan section and it works:

root@puppet-test:~# dpkg -l puppet
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================-============-=================================
ii puppet 5.5.22-4ubuntu0.1 all configuration management system
root@puppet-test:~# cat test.pp
file { '/tmp/link-to-motd':
   ensure => 'link',
   target => '/etc/motd',
}
root@puppet-test:~# puppet apply test.pp
Notice: Compiled catalog for puppet-test.lxd in environment production in 0.01 seconds
Notice: /Stage[main]/Main/File[/tmp/link-to-motd]/ensure: created
Notice: Applied catalog in 0.01 seconds
root@puppet-test:~# ls /tmp/link-to-motd -las
0 lrwxrwxrwx 1 root root 9 May 9 20:15 /tmp/link-to-motd -> /etc/motd

The issue you are facing regarding to_sym is another thing, I am trying to identify what is happening. I took what Serge did (comment #11) but I was not able to reproduce the issue:

root@puppet-test:~# cat test.pp
file { '/etc/localtime':
                ensure => 'link',
                target => $osfamily ? {
                        'Debian' => '/usr/share/zoneinfo/posix/America/Los_Angeles',
                        default => '/usr/share/zoneinfo/America/Los_Angeles',
  }
}
root@puppet-test:~# puppet apply test.pp
Notice: Compiled catalog for puppet-test.lxd in environment production in 0.01 seconds
Notice: /Stage[main]/Main/File[/etc/localtime]/target: target changed '/usr/share/zoneinfo/Etc/UTC' to '/usr/share/zoneinfo/posix/America/Los_Angeles'
Notice: Applied catalog in 0.02 seconds
root@puppet-test:~# ls -las /etc/localtime
0 lrwxrwxrwx 1 root root 45 May 9 13:21 /etc/localtime -> /usr/share/zoneinfo/posix/America/Los_Angeles

How exactly are you running your manifests? I'd need to reproduce the problem locally to keep investigating. Ideally, the simplest manifest needed to trigger this issue.

This bug was fixed in the package puppet - 5.5.22-4ubuntu1

---------------
puppet (5.5.22-4ubuntu1) kinetic; urgency=medium

  * Backport upstream patches (LP: #1969939):
    - d/p/0017-PUP-10537-keyword-param-as-last-arg.patch: Add patch to fix
      ruby 3.0 incompatibilities.
    - d/p/0018-PUP-11045-ruby-openssl-now-sets-store_context.error.patch: Add
      upstream patch to better support openssl 3.

 -- Lucas Kanashiro <email address hidden> Fri, 29 Apr 2022 16:07:16 -0300

Hi! I did several tests and looks like the symlink problem is resolved.
However still see those error messages listed above.

Since no one gave me a way yet to reproduce the mentioned errors locally, and the originally reported bug (the symlink issue) is fixed, I am marking the verification as done.

Please, file a new bug with reproduction steps for the error you all mentioned, so I can investigate it further.

As an SRU team member, it's unclear to me whether the errors reported in this verification are introduced by this update or are merely existing problems that the code now hits because it's getting further.

It's also not clear to me what the effect of these errors is - if these are just harmless warnings printed, that's ok, but if they are exposing real problems that's an issue. Even more if they're exposing problems that might hit users that are not already broken by this bug.

Can we please get an analysis of what - if anything - is broken here?

Chris, I think the main functionality works OK (i.e. it creates a link),
but I'm confused what are those errors and what do they mean?
Also I'm going to continue with more configuration and see if we get any
more errors or all this is fine.

ср, 25 трав. 2022 р. о 06:16 Chris Halse Rogers <email address hidden>
пише:

> As an SRU team member, it's unclear to me whether the errors reported in
> this verification are introduced by this update or are merely existing
> problems that the code now hits because it's getting further.
>
> It's also not clear to me what the effect of these errors is - if these
> are just harmless warnings printed, that's ok, but if they are exposing
> real problems that's an issue. Even more if they're exposing problems
> that might hit users that are not already broken by this bug.
>
> Can we please get an analysis of what - if anything - is broken here?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1969939
>
> Title:
> puppet 5 package incompatible with ruby 3 / ubuntu 22.04
>
> Status in puppet package in Ubuntu:
> Fix Released
> Status in puppet source package in Jammy:
> Fix Committed
> Status in puppet source package in Kinetic:
> Fix Released
> Status in puppet package in Debian:
> New
>
> Bug description:
> [Impact]
>
> puppet crashes if one tries to create/ensure symlinks via a puppet
> manifest. I believe this is an feature broadly used by our users.
>
> [Test Plan]
>
> $ lxc launch ubuntu-daily:jammy puppet-test
> $ lxc shell puppet-test
> # apt update && apt install -y puppet
> # cat <<EOF > hellopuppet.pp
> file { '/tmp/link-to-motd':
> ensure => 'link',
> target => '/etc/motd',
> }
> EOF
> # puppet apply hellopuppet.pp
> Notice: Compiled catalog for k1.lxd in environment production in 0.01
> seconds
> Error: Could not set 'link' on ensure: wrong number of arguments (given
> 3, expected 2) (file: /home/ubuntu/hellopuppet.pp, line: 1)
> Error: Could not set 'link' on ensure: wrong number of arguments (given
> 3, expected 2) (file: /home/ubuntu/hellopuppet.pp, line: 1)
> Wrapped exception:
> wrong number of arguments (given 3, expected 2)
> Error: /Stage[main]/Main/File[/tmp/link-to-motd]/ensure: change from
> 'absent' to 'link' failed: Could not set 'link' on ensure: wrong number of
> arguments (given 3, expected 2) (file: /home/ubuntu/hellopuppet.pp, line: 1)
> Notice: Applied catalog in 0.02 seconds
>
> With the proposed fix the crash should not happen and the link
> created.
>
> [Where problems could occur]
>
> The risk of the patch introduce a bug is quite low, what might happen
> is some more occurrences of this issue throughout the codebase. I went
> through upstream git repo and issue tracker and I think this is
> enough.
>
> [Original description]
>
> Hi,
>
> the puppet package that comes with ubuntu 22.04 is not completely
> working under ubuntu 22.04, since it is (outdated) puppet 5 which is
> not compatible with ruby versions above ruby 2.7, where ubuntu 22.04
> comes with ruby 3.
>
> The problem seems to be just a minor proble...

As you can see in comment #17, I have not been able to reproduce the error people are reporting, so I can't help with this analysis. From what I understood, this is not crashing anything (like it is right now when someone tries to create a symlink).

Hi Lucas, the new to_sym error seems to be related to use of the File resource with `ensure => file` instead of link. Unless there's another change introduced in 5.5.22-4ubuntu0.1, it seems like it has to be caused by your patch.

Here's my test scenario that you should be able to use to reproduce:

issue.pp:

```
class issue
{
    file { '/tmp/issue':
        ensure => file,
        source => '/etc/issue',
    }
}

include issue
```

# Using puppet (5.5.22-4):

% puppet apply issue.pp
Notice: Compiled catalog for dev-vagrant22 in environment production in 0.01 seconds
Notice: /Stage[main]/Issue/File[/tmp/issue]/ensure: defined content as '{md5}d5faf84af6529c6e6a364f164e953ff6'
Notice: Applied catalog in 0.01 seconds

# Using puppet (5.5.22-4ubuntu0.1):

% puppet apply issue.pp
Notice: Compiled catalog for dev-vagrant22 in environment production in 0.01 seconds
Error: /Stage[main]/Issue/File[/tmp/issue]: Could not evaluate: Could not retrieve file metadata for file:///etc/issue: undefined method `to_sym' for nil:NilClass
Notice: Applied catalog in 0.01 seconds

Thanks for your help fixing this, Lucas!

My test is also unnecessarily complex -- there's no needs for the class + include:

file { '/tmp/issue':
    ensure => file,
    source => '/etc/issue',
}

Thanks for the reproducer Mat! I can reproduce it locally now, and this is also impacting kinetic. I'll be working on it.

If I understand correctly (based on comment #24), does this mean that the to_sym error seems to be a regression introduced by the patch currently in jammy-proposed, right? In this case I think we need to either switch this to verification-failed or at least to verification-needed. Let me switch those to verification-needed and please give an info whether this update is still something we want to consider for -updates.

Hi all,

I prepared a patch myself to fix this. Could you please test the package in this PPA? https://launchpad.net/~lucaskanashiro/+archive/ubuntu/puppet/

This is a workaround and I'd like to make sure this is not impacting anything else. Please, report back with your findings. Once we get enough feedback I'll move forward to push this into the archive.

Thanks Lucas. LGTM. We tried the PPA. No errors during puppet apply. File resources, both symlinks and regular files, are being created as expected. Our full 5kloc puppet manifest applied without any puppet errors, and we haven't noticed any other regressions. If you have any specific functionality you like us to double check, please let me know.

Thank you very much, your help to fix the 5.5 agent package (we noticed the symlink error, too) is really appreciated.
I was very happy that we can just use the puppet 5.5 agent package straight from Ubuntu 22 to use our well established puppet 5.5 server so we do not have to setup an all new puppet 7 server.

Thanks for the feedback everyone. I'll give a chance to others to test it over the weekend, if I get no negative feedback until Monday I'll be uploading this fix to kinetic and SRU it to jammy.

Since no negative feedback was reported here I went ahead and uploaded the patch I provided in my PPA last week. This is fixed in kinetic-proposed (version 5.5.22-4ubuntu2) and I uploaded version 5.5.22-4ubuntu0.2 to jammy-unapproved.

SRU team: please take a look at this new version, and if everything looks OK let's land it in jammy-proposed and ask affected people to test it again.

This bug was fixed in the package puppet - 5.5.22-4ubuntu2

---------------
puppet (5.5.22-4ubuntu2) kinetic; urgency=medium

  * Add patch to set file's link attribute to manage by default (LP: #1969939)

 -- Lucas Kanashiro <email address hidden> Tue, 31 May 2022 11:36:34 -0300

Hello Hadmut, or anyone else affected,

Accepted puppet into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/puppet/5.5.22-4ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

With the puppet package version 5.5.22-4
from
http://de.archive.ubuntu.com/ubuntu jammy/universe

the following puppet manifest did not work with Ubuntu 22.04 LTS jammy:

file {'/etc/alternatives/editor':
  ensure => link,
  owner => 'root',
  group => 'root',
  target => '/usr/bin/vim.basic',
  require => Package['vim'],
}

There was an error message like in OP.

This bug was resolved with puppet package version 5.5.22-4ubuntu0.2
from
http://de.archive.ubuntu.com/ubuntu jammy-proposed/universe

With that version the symlink gets created/changed like it was the case with e.g. Ubuntu 18.0 Bionic:

sudo rm /etc/alternatives/editor

sudo puppet agent -t

Info: Applying configuration version '1234567'
Notice: /Stage[main]/linux_base::Vim/File[/etc/alternatives/editor]/ensure: created (corrective)
Notice: Applied catalog in 27.46 seconds

So puppet package version 5.5.22-4ubuntu0.2 is the bugfix for us.

I hope that version will be published in the official jammy or jammy-security repository, so we can use the standard ubuntu repo.

P.S. We have 72 puppet manifest files which all run fine with agent 5.5.22-4ubuntu0.2 on Ubuntu 22 Jammy as they run with agent 5.5.2 on Ubuntu 18 Bionic, using the same 5.5 puppet server.

Thanks for testing the version in jammy-proposed Miranda, and confirming it is working as expected.

I did my tests as well, all is fine! Thank you for help.

вт, 14 черв. 2022 р., 15:50 користувач Lucas Kanashiro <
<email address hidden>> пише:

> Thanks for testing the version in jammy-proposed Miranda, and confirming
> it is working as expected.
>
> ** Tags removed: verification-needed verification-needed-jammy
> ** Tags added: verification-done verification-done-jammy
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1969939
>
> Title:
> puppet 5 package incompatible with ruby 3 / ubuntu 22.04
>
> Status in puppet package in Ubuntu:
> Fix Released
> Status in puppet source package in Jammy:
> Fix Committed
> Status in puppet source package in Kinetic:
> Fix Released
> Status in puppet package in Debian:
> New
>
> Bug description:
> [Impact]
>
> puppet crashes if one tries to create/ensure symlinks via a puppet
> manifest. I believe this is an feature broadly used by our users.
>
> [Test Plan]
>
> $ lxc launch ubuntu-daily:jammy puppet-test
> $ lxc shell puppet-test
> # apt update && apt install -y puppet
> # cat <<EOF > hellopuppet.pp
> file { '/tmp/link-to-motd':
> ensure => 'link',
> target => '/etc/motd',
> }
> EOF
> # puppet apply hellopuppet.pp
> Notice: Compiled catalog for k1.lxd in environment production in 0.01
> seconds
> Error: Could not set 'link' on ensure: wrong number of arguments (given
> 3, expected 2) (file: /home/ubuntu/hellopuppet.pp, line: 1)
> Error: Could not set 'link' on ensure: wrong number of arguments (given
> 3, expected 2) (file: /home/ubuntu/hellopuppet.pp, line: 1)
> Wrapped exception:
> wrong number of arguments (given 3, expected 2)
> Error: /Stage[main]/Main/File[/tmp/link-to-motd]/ensure: change from
> 'absent' to 'link' failed: Could not set 'link' on ensure: wrong number of
> arguments (given 3, expected 2) (file: /home/ubuntu/hellopuppet.pp, line: 1)
> Notice: Applied catalog in 0.02 seconds
>
> With the proposed fix the crash should not happen and the link
> created.
>
> [Where problems could occur]
>
> The risk of the patch introduce a bug is quite low, what might happen
> is some more occurrences of this issue throughout the codebase. I went
> through upstream git repo and issue tracker and I think this is
> enough.
>
> [Original description]
>
> Hi,
>
> the puppet package that comes with ubuntu 22.04 is not completely
> working under ubuntu 22.04, since it is (outdated) puppet 5 which is
> not compatible with ruby versions above ruby 2.7, where ubuntu 22.04
> comes with ruby 3.
>
> The problem seems to be just a minor problem, it can't set logical
> links :
>
> Could not set 'link' on ensure: wrong number of arguments (given 3,
> expected 2)
>
> The bug is known and discussed in upstream debian under
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009643
>
> but they do believe that it is not worth wasting time to fix that old
> version and rather update to puppet 7.
>
> From what they tell, it takes at least puppet 7.8 to run under ruby 3.
>
> So as a workaround, one could use the regular puppet gem (whi...

This bug was fixed in the package puppet - 5.5.22-4ubuntu0.2

---------------
puppet (5.5.22-4ubuntu0.2) jammy; urgency=medium

  * d/p/0018-Set-file-links-attribute-to-manage-by-default.patch: the 'links'
    attribute of a file object was set to 'nil' by default which was causing a
    crash when the to_sym function was called. Now, it is set to 'manage' by
    default (which is the upstream documented default value) and the crash is
    fixed (LP: #1969939).

 -- Lucas Kanashiro <email address hidden> Tue, 31 May 2022 11:44:35 -0300

The verification of the Stable Release Update for puppet has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.