[queens] innodb log file size is insufficient

Bug #1831875 reported by Michele Baldessari
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
High
Michele Baldessari

Bug Description

Sometimes on queens we can get the following error:
"""
2018-08-03 15:03:35.898 1544 ERROR oslo_messaging.rpc.server DBError: (pymysql.err.InternalError) (1118, u'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.') [SQL: u'UPDATE action_executions_v2 SET updated_at=%(updated_at)s, state=%(state)s, accepted=%(accepted)s, output=%(output)s WHERE action_executions_v2.id = %(action_executions_v2_id)s'] [parameters: {'output': '{"result": {"log_path": "/tmp/ansible-mistral-actionIMP4Az/ansible.log", "stderr": "ansible-playbook 2.4.3.0\\n config file = /usr/share/ceph-ansibl ... (11486221 characters truncated) ... statements should not include jinja2 templating delimiters\\nsuch as {{ }} or {% %}. Found: {{ groups.get(mgr_group_name, []) | length > 0\\n}}\\n"}}', 'state': 'SUCCESS', 'accepted': 1, 'updated_at': datetime.datetime(2018, 8, 3, 19, 3, 35), 'action_executions_v2_id': u'f9170825-5b9c-4d41-b528-4b029a78796b'}] (Background on this error at: http://sqlalche.me/e/2j85)
"""

This is due to ansible output being stored in the db and it happens only in queens.
In fact starting with rocky we do not store this output in the DB any longer because we switched to config-dwonload workflow:
https://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml#L479
 which has a parameter 'trash_output' which is set to true by default, meaning that we will not be storing the output in the db any longer.

In order to mitigate the problem in queens, we should increase the innodb log file size by default on the undercloud.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/663595

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (stable/queens)
Download full text (4.0 KiB)

Reviewed: https://review.opendev.org/663595
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=a098fbc64d6442317352913bc935da812bd32b9e
Submitter: Zuul
Branch: stable/queens

commit a098fbc64d6442317352913bc935da812bd32b9e
Author: Michele Baldessari <email address hidden>
Date: Thu Jun 6 12:42:31 2019 +0200

    [queens-only] Increase the innodb file size by default

    Sometimes on queens we can get the following error:
    """
    2018-08-03 15:03:35.898 1544 ERROR oslo_messaging.rpc.server DBError: (pymysql.err.InternalError) (1118, u'The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.') [SQL: u'UPDATE action_executions_v2 SET updated_at=%(updated_at)s, state=%(state)s, accepted=%(accepted)s, output=%(output)s WHERE action_executions_v2.id = %(action_executions_v2_id)s'] [parameters: {'output': '{"result": {"log_path": "/tmp/ansible-mistral-actionIMP4Az/ansible.log", "stderr": "ansible-playbook 2.4.3.0\\n config file = /usr/share/ceph-ansibl ... (11486221 characters truncated) ... statements should not include jinja2 templating delimiters\\nsuch as {{ }} or {% %}. Found: {{ groups.get(mgr_group_name, []) | length > 0\\n}}\\n"}}', 'state': 'SUCCESS', 'accepted': 1, 'updated_at': datetime.datetime(2018, 8, 3, 19, 3, 35), 'action_executions_v2_id': u'f9170825-5b9c-4d41-b528-4b029a78796b'}] (Background on this error at: http://sqlalche.me/e/2j85)
    """

    By increasing the innodb log file size from 50MB to 256MB by default we should
    mitigate this problem considerably (rocky and onwards are not affected as config-download does
    not store the output in the DB).

    Tested as follows:
    1) New undercloud deploy with this patch -> correct innodb file size (256M)

    2) Redeploy of an existing undercloud, hereby increasing (from 50MB as default to 256M)

    Before:
    [root@undercloud-0 ~]# mysql -e "show variables like 'innodb_log_file_size';"
    +----------------------+----------+
    | Variable_name | Value |
    +----------------------+----------+
    | innodb_log_file_size | 50331648 |
    +----------------------+----------+

    After:
    [root@undercloud-0 ~]# mysql -e "show variables like 'innodb_log_file_size';"
    +----------------------+-----------+
    | Variable_name | Value |
    +----------------------+-----------+
    | innodb_log_file_size | 268435456 |
    +----------------------+-----------+

    mariadb indeed has increased the log file correctly:
    2019-06-06 6:03:20 139969388783808 [Warning] InnoDB: Resizing redo log from 2*3072 to 2*16384 pages, LSN=40967264
    2019-06-06 6:03:20 139969388783808 [Warning] InnoDB: Starting to delete and rewrite log files.
    2019-06-06 6:03:20 139969388783808 [Note] InnoDB: Setting log file ./ib_logfile101 size to 256 MB InnoDB: Progress in MB: 100 200
    2019-06-06 6:03:21 139969388783808 [Note] InnoDB: Setting log file ./ib_logfile1 size to 256 MB

    And the files are correctly resized:
    -rw-rw----. 1 mysql mysql 268435456 Jun 6 06:14 ib_logfile0
    -rw-rw----. 1 mysql mysql...

Read more...

tags: added: in-stable-queens
Changed in tripleo:
milestone: train-1 → train-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 8.4.8

This issue was fixed in the openstack/instack-undercloud 8.4.8 release.

Changed in tripleo:
milestone: train-2 → train-3
Changed in tripleo:
milestone: train-3 → ussuri-1
Changed in tripleo:
milestone: ussuri-1 → ussuri-2
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.