fuel-agent: wrong amount of arguments for logger fails provisioning

Bug #1597697 reported by Alexander Gordeev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Alexey Stupnikov
Mitaka
Won't Fix
High
Alexey Stupnikov

Bug Description

apparently, the fix for https://bugs.launchpad.net/fuel/+bug/1537699 contains typo in logger call.

https://github.com/openstack/fuel-agent/blob/5809a3fea96b76386ddc4ccd367c21c00aa8337d/fuel_agent/manager.py#L511-L512

this is critical and could lead to provisioning failure.

Traceback (most recent call last):
  File \"/usr/lib/python2.7/logging/__init__.py\", line 861, in emit
    msg = self.format(record)
  File \"/usr/lib/python2.7/logging/__init__.py\", line 734, in format
    return fmt.format(record)
  File \"/usr/lib/python2.7/dist-packages/oslo_log/formatters.py\", line 297, in format
    return logging.Formatter.format(self, record)
  File \"/usr/lib/python2.7/logging/__init__.py\", line 465, in format
    record.message = record.getMessage()
  File \"/usr/lib/python2.7/logging/__init__.py\", line 329, in getMessage
    msg = msg % self.args
TypeError: not enough arguments for format string
Logged from file manager.py, line 519

tags: added: area-python
Changed in fuel:
milestone: none → 10.0
Changed in fuel:
assignee: nobody → Alexey Stupnikov (astupnikov)
Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

The issue is reproducible only on custom 10.0 build with ubuntu xenial which probably contains newer version of oslo_log library.

All other fuel releases haven't been affected by it due to the fact that fuel-agent uses obsolete and stailed part of oslo_incubator's log.py which behaves in the following way:

The message will not be logged because a string formatting exception will occur. it just silently ignores the error and continues to work further with no visible side effects.

Changed in fuel:
status: New → Confirmed
Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

When I have fixed reported issue I have found another issue:
fuel_agent.manager [-] Path exists. Trying to sync all files from /tmp/tmpZWXSft/usr/ to /tmp/tmp8paYGQ
fuel_agent.utils.utils [-] Trying to execute command: rsync -avH /tmp/tmpZWXSft/usr// /tmp/tmp8paYGQ

When manager is working with mount point itself, it already has trailing "/" and we don't have to add another one. Though it is not breaking anything, it is still better to fix this one.

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Steps to reproduce:
1. Create new env
2. Assign 1 controller to deploy
3. Increase base system space via Web UI to account for extra LV space (for ex. add 10240MB to base, take it from Image)
3. `fuel provisioning default --env 1`
4. edit the node yaml - make root LV smaller by 10G, add an extra LV 'log' with 10G size:
    - _allocate_size: min
      id: os
      label: Base System
      min_size: 19456
      type: vg
      volumes:
      - file_system: ext4
        mount: /
        name: root
- size: 30720
+ size: 20480
        type: lv
+ - file_system: ext4
+ mount: /usr/
+ name: usr
+ size: 10240
+ type: lv
      - file_system: swap
        mount: swap
        name: swap
        size: 4096
        type: lv
5. `fuel provisioning upload --env 1`
6. start deployment

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Correct log messages:
2016-06-30 17:50:34 INFO fuel_agent.utils.utils [-] Trying to execute command: rsync -avH /tmp/tmpIBAT4v/usr/ /tmp/tmpCrtoQ9
2016-06-30 17:50:34 INFO fuel_agent.manager [-] Path exists. Trying to sync all files from /tmp/tmpIBAT4v/usr/ to /tmp/tmpCrtoQ9
2016-06-30 17:50:34 INFO fuel_agent.manager [-] Trying to check if path /tmp/tmpIBAT4v/usr/ exist

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/336147

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/336149

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

This issue will be fixed in 7.0 and 8.0 branches in bug #1537699

Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

definitely low for anything older than 10.0

so, it may be closed as won't fix.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (master)

Reviewed: https://review.openstack.org/336147
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=e6a31c84560713c374d65a6315249a65c94e7ae3
Submitter: Jenkins
Branch: master

commit e6a31c84560713c374d65a6315249a65c94e7ae3
Author: Alexey Stupnikov <email address hidden>
Date: Thu Jun 30 20:58:27 2016 +0300

    Fix logger issue in move_files_to_their_places function

    A typo in manager.py was fixed. This typo doesn't break
    anything when old enough oslo.log module is used, but it
    breaks deployment process when newer OS is used to build
    bootstrap image.

    A minor issue with path build process was also fixed.

    Change-Id: I97616ef0bfbe0f7ffc6e625e4bf7c7ffde1a1919
    Closes-Bug: #1597697

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Abandoned stable/mitaka patch. Added fixes to stable/7.0 and stable/8.0 patches on review.

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Returned old milestone, original change was made because of my lack of experience with MOS9 workflow.

Revision history for this message
Stanislaw Bogatkin (sbogatkin) wrote :

This problem broke our provisioning today at verifying provisioning as a graph. Reopen.

Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

It turned out, that fuel-agent just silently barfs this message of traceback to stderr. It doesn't affect the execution/operation anyhow. Provisioning went smooth with no errors at all and returned zero exit code as well.

So, moving back to won't fix for 9.1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-agent 10.0.0rc1

This issue was fixed in the openstack/fuel-agent 10.0.0rc1 release candidate.

Revision history for this message
Stanley@Linux Simba (linuxsimba) wrote :

This issue affects Ironic conductor debugging in MOS 9.1. It causes ironic server instance deployment to fail if you turn on debug=True on /etc/ironic/ironic.conf on the Ironic conductor.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-agent 10.0.0

This issue was fixed in the openstack/fuel-agent 10.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-agent (stable/mitaka)

Change abandoned by Joshua Hesketh (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/336149
Reason: This branch (stable/mitaka) is at End Of Life

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/336149
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Closing as won't fix for MOS9 as this was a customization request for a very specific case, that is not needed anymore.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.