Galera

gcache.page. files not removed fast enough for some workloads

Bug #1488530 reported by Przemek
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Ivan Suzdal
Fuel for OpenStack 9.0
6.1.x
Invalid
High
Rodion Tikunov
Fuel for OpenStack 6.1-updates
7.0.x
Invalid
High
Rodion Tikunov
Fuel for OpenStack 7.0-updates
8.0.x
Fix Released
High
Ivan Suzdal
Fuel for OpenStack 8.0
Galera
New
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Confirmed
High
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC future-5.6

Bug Description

This is copy of initial report: https://github.com/codership/galera/issues/317

For workloads with huge transactions it may happen that gcache.page.xxxx files will fill all the available disk space as due to current removal algorithm, oldest files may remain pretty long.

Here is a quickly reproducible test case:

perl -e '$s="$s\x31"; for my $i (0..26214400) { print $s; } ' > /tmp/blob

use test
CREATE TABLE blob1 (
id int(11) NOT NULL,
a tinyint(4) DEFAULT NULL,
big longblob,
PRIMARY KEY (id)
) ENGINE=InnoDB;
insert into blob1 values (1,1,LOAD_FILE('/tmp/blob'));

for i in {1..50}; do mysql test -e "UPDATE blob1 SET a=$i,big=LOAD_FILE('/tmp/blob')"; done

Tags: i57958
Raghavendra D Prabhu (raghavendra-prabhu)
Changed in percona-xtradb-cluster:
milestone: none → 5.6.25-25.12
Przemek (pmalkowski)
Changed in percona-xtradb-cluster:
status: New → Confirmed
Raghavendra D Prabhu (raghavendra-prabhu)
Changed in percona-xtradb-cluster:
milestone: 5.6.25-25.12 → future-5.6
Krunal Bauskar (krunal-bauskar)
Changed in percona-xtradb-cluster:
importance: Undecided → High
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → MOS Linux (mos-linux)
milestone: none → 9.0
Ivan Suzdal (isuzdal)
Changed in fuel:
assignee: MOS Linux (mos-linux) → Ivan Suzdal (isuzdal)
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to packages/trusty/galera (master)

Related fix proposed to branch: master
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/16432

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to packages/trusty/galera (master)

Reviewed: https://review.fuel-infra.org/16432
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: e7558b9c8ae2893ebef62cd5dd213b737c0b4f36
Author: Ivan Suzdal <email address hidden>
Date: Mon Jan 25 12:55:55 2016

Fix for gcache.page files removal.

Change-Id: Ia74799e9ea55a04a23cc0f13bab638522d1d08ec
Related-Bug: #1488530

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to packages/trusty/galera (8.0)

Related fix proposed to branch: 8.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/16499

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to packages/trusty/galera (8.0)

Reviewed: https://review.fuel-infra.org/16499
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: 8acb72b37f8efa0b7a0911e4fd399dfcf8dadcd7
Author: Ivan Suzdal <email address hidden>
Date: Wed Jan 27 18:09:58 2016

Fix for gcache.page files removal.

Change-Id: Ia74799e9ea55a04a23cc0f13bab638522d1d08ec
Related-Bug: #1488530
(cherry picked from commit e7558b9c8ae2893ebef62cd5dd213b737c0b4f36)

Ivan Suzdal (isuzdal)
Changed in fuel:
status: Confirmed → Fix Committed
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Verified om MOS 8.0 RC1:

mysql> use test
ERROR 1049 (42000): Unknown database 'test'
mysql> CREATE DATABASE test
    -> ;
Query OK, 1 row affected (0.00 sec)

mysql> use test
Database changed
mysql> CREATE TABLE blob1 (
    -> id int(11) NOT NULL,
    -> a tinyint(4) DEFAULT NULL,
    -> big longblob,
    -> PRIMARY KEY (id)
    -> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.03 sec)

mysql> insert into blob1 values (1,1,LOAD_FILE('/tmp/blob'));
ERROR 1118 (42000): The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.

So, the size of this file was reduced.

issue not reproduced after
root@node-1:~# for i in {1..50}; do mysql test -e "UPDATE blob1 SET a=$i,big=LOAD_FILE('/tmp/blob')"; done
root@node-1:~# for i in {1..5000}; do mysql test -e "UPDATE blob1 SET a=$i,big=LOAD_FILE('/tmp/blob')"; done
root@node-1:~#

Moved to Fix Released.

Revision history for this message
Rodion Tikunov (rtikunov) wrote :

In 7.0 galera updated to galera-3 and the bug can not be reproduced.

Revision history for this message
Rodion Tikunov (rtikunov) wrote :

Could not reproduce for 6.1

Revision history for this message
Alexander Gubanov (ogubanov) wrote :

Verified on MOS 9.0 (build 201) - can't reproduce.
Details: http://pastebin.com/fiKp86nA

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
pippo (claudio-enjoy) wrote :

same issue with :

Percona-XtraDB-Cluster-client-56-5.6.35-26.20.2.el6.x86_64
Percona-XtraDB-Cluster-shared-56-5.6.35-26.20.2.el6.x86_64
Percona-XtraDB-Cluster-56-5.6.35-26.20.2.el6.x86_64
percona-toolkit-2.2.15-2.noarch
Percona-XtraDB-Cluster-galera-3-3.20-2.el6.x86_64
percona-xtrabackup-2.3.5-1.el6.x86_64
Percona-XtraDB-Cluster-server-56-5.6.35-26.20.2.el6.x86_64

both configuration generate 300+ gcache.page.000* logs

1 wsrep_provider_options = gcache.size=2G;gcache.keep_pages_size=100M;gcache.keep_pages_count=20
2 wsrep_provider_options = gcache.size=2G;gcache.keep_pages_size=2GB;gcache.keep_pages_count=10

any suggestion ?

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1025

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.