slaves crashed under randgen transactions test

Bug #1136966 reported by Vadim Tkachenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Incomplete
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Expired
Undecided
Unassigned

Bug Description

In 6 nodes cluster randgen load was put to a master
and at some point all 5 slaves crashed with an identical error:

130228 22:55:23 [ERROR] Slave SQL: Could not execute Update_rows event on table test.m; Can't find record in 'm', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log FIRST, end_log_pos 1162, Error_code: 1032
130228 22:55:23 [Warning] WSREP: RBR event 2 Update_rows apply warning: 137, 1529610
130228 22:55:23 [ERROR] WSREP: Failed to apply trx: source: 6aac4226-8221-11e2-0800-9c7f0e75c63c version: 2 local: 0 state: APPLYING flags: 1 conn_id: 41 trx_id: 4085687 seqnos (l: 8661, g: 1529610, s: 1529609, d: 1529573, ts: 1362110123846395570)
130228 22:55:23 [ERROR] WSREP: Failed to apply app buffer: «&0Q^S, seqno: 1529610, status: WSREP_FATAL
         at galera/src/replicator_smm.cpp:apply_wscoll():49
         at galera/src/replicator_smm.cpp:apply_trx_ws():120

randgen command is:

perl gentest.pl \
     --dsn=dbi:mysql:host=208.88.225.244:port=3306:user=root:database=test \
     --gendata=conf/transactions/transactions.zz \
     --grammar=conf/transactions/transactions.yy \
     --threads=5 \
     --queries=10000000000 \
     --duration=300 \
     --seed=time \
     --reporter=Shutdown,Backtrace,QueryTimeout \
     --mask=0 --mask-level=0

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Vadim, with randgen 2.2.0 I get the following:

unable to open specification file 'conf/transactions/transactions.zz': No such file or directory at gentest.pl line 175

What am I missing? Are conf/transactions.* good enough?

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Also, was it a multi-master or master-slave test?

Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

My setup is working just after these steps

1. bzr branch lp:randgen

2. Make sure to have the many (Perl modules etc.) dependencies installed:
    https://github.com/RQG/RQG-Documentation/wiki/RandomQueryGeneratorQuickStart#wiki-Prerequisites

in conf/transactions/ I have

 ls
combinations.yy repeatable_read.yy transactions-flat.yy transactions.zz
combinations.zz transaction_durability.yy transactions.yy

I put load only to single master and rest 5 nodes are "slaves"

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Vadim, followed your instructions, but could not reproduce with the current lp:~codership/codership-mysql/5.5-23 head. So

1) Did you run it against codership-mysql codebase and what was the revision?
2) Is it reproducible?
3) Do you have an error log from a crashed slave?

Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

Indeed I can't repeat it with lp:~codership/codership-mysql/5.5-23 head

Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

Funny (or not) I can't repeat now with PXC also.

But I think I know what is difference.
Now I am using an empty, fresh-created database.

It seems when I was getting a crash, there was some conflict with existing databases,
but for now I can't repeat it.

Changed in percona-xtradb-cluster:
status: New → Incomplete
Changed in codership-mysql:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for Percona XtraDB Cluster because there has been no activity for 60 days.]

Changed in percona-xtradb-cluster:
status: Incomplete → Expired
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1303

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.