MMM could not fail-over when server hang

I hit this problem 2 times. I think it's related to the fact that after crash, system can be still responsive.

In source code (Monitor.pm):

                if ($state eq 'ONLINE') {
                        # ONLINE -> HARD_OFFLINE
                        unless ($ping && $mysql) {

Shouldn't it be OR instead of AND?

And one more thing:

sub move_role()

        # Assign role to new host
        $roles->set_role($role, $ip, $host);

        # Notify old host (if is_active_master_role($role) this will make the host non writable)
        $monitor->send_agent_status($old_owner);

        # Notify slaves (this will make them switch the master)
        $monitor->notify_slaves($host) if ($roles->is_active_master_role($role));

        # Notify new host (if is_active_master_role($role) this will make the host writable)
        $monitor->send_agent_status($host);

and:

        # Finally send command
        my $ret = $agent->cmd_set_status($master);

        unless ($ret) {
                # If ping is down, nothing will be send to agent. So this doesn't indicate that the agent is down.
                my $checks = $self->checks_status;
                if ($checks->ping($host) && !$agent->agent_down()) {
                        FATAL "Can't reach agent on host '$host'";
                        $agent->agent_down(1);
                }
        }
        elsif ($agent->agent_down) {
                FATAL "Agent on host '$host' is reachable again";
                $agent->agent_down(0);
        }
        return $ret;
}

This is another possible reason (self explained above).

Logs from problematic period of time:

System arch:
- MONITORING SERVER
- 2 machines in master-master pair (db12/db13)
- additional slave (db14)

MONITORING SERVER:
2010/08/13 13:16:35 FATAL Can't reach agent on host 'db12'
2010/08/13 13:20:27 ERROR Check 'ping' on 'db12' has failed for 242 seconds! Message: ERROR: Could not ping 10.16.214.132
2010/08/13 13:20:32 FATAL State of host 'db12' changed from ONLINE to HARD_OFFLINE (ping: not OK, mysql: OK)
2010/08/13 13:20:32 INFO Removing all roles from host 'db12':
2010/08/13 13:20:32 INFO Removed role 'writer(10.17.88.140)' from host 'db12'
2010/08/13 13:20:32 INFO Orphaned role 'writer(10.17.88.140)' has been assigned to 'db13'
..
(nothing here, waiting for db12 to become online)

2010/08/13 14:16:27 FATAL Agent on host 'db12' is reachable again

db12:
--
(this one crashed, no MMM logs)

db13:

2010/08/13 14:16:24 INFO We have some new roles added or old rules deleted!
2010/08/13 14:16:24 INFO Added: writer(10.17.88.140)
(this is exactly when db12 appeared online)

db14:
2010/08/13 13:20:32 INFO Changing active master to 'db13'

I didn't drill into source code but "(ping: not OK, mysql: OK)" and "# If ping is down, nothing will be send to agent. So this doesn't indicate that the agent is down." sounds like possible reason.

Hi, I found a cause of this situation.
You can find below source code in Monitor.pm
...
- # Finally send command
- my $ret = $agent->cmd_set_status($master);
...

Periodically, MMM Monitor send status file to all alive agent, but there is no timeout logic.
Even after a node crash, MMM Monitor try sending status file to crashed node and finally blocked.
So, I added a timeout logic.
I attached patch file, check it out. :-)

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Piotr Biel
Sent: Monday, September 06, 2010 11:14 PM
To: <email address hidden>
Subject: [Bug 590623] Re: MMM could not fail-over when server hang

I hit this problem 2 times. I think it's related to the fact that after crash, system can be still responsive.

In source code (Monitor.pm):

                if ($state eq 'ONLINE') {
                        # ONLINE -> HARD_OFFLINE
                        unless ($ping && $mysql) {

Shouldn't it be OR instead of AND?

And one more thing:

sub move_role()

        # Assign role to new host
        $roles->set_role($role, $ip, $host);

        # Notify old host (if is_active_master_role($role) this will make the host non writable)
        $monitor->send_agent_status($old_owner);

        # Notify slaves (this will make them switch the master)
        $monitor->notify_slaves($host) if ($roles->is_active_master_role($role));

        # Notify new host (if is_active_master_role($role) this will make the host writable)
        $monitor->send_agent_status($host);

and:

        # Finally send command
        my $ret = $agent->cmd_set_status($master);

        unless ($ret) {
                # If ping is down, nothing will be send to agent. So this doesn't indicate that the agent is down.
                my $checks = $self->checks_status;
                if ($checks->ping($host) && !$agent->agent_down()) {
                        FATAL "Can't reach agent on host '$host'";
                        $agent->agent_down(1);
                }
        }
        elsif ($agent->agent_down) {
                FATAL "Agent on host '$host' is reachable again";
                $agent->agent_down(0);
        }
        return $ret;
}

This is another possible reason (self explained above).

Logs from problematic period of time:

System arch:
- MONITORING SERVER
- 2 machines in master-master pair (db12/db13)
- additional slave (db14)

MONITORING SERVER:
2010/08/13 13:16:35 FATAL Can't reach agent on host 'db12'
2010/08/13 13:20:27 ERROR Check 'ping' on 'db12' has failed for 242 seconds! Message: ERROR: Could not ping 10.16.214.132
2010/08/13 13:20:32 FATAL State of host 'db12' changed from ONLINE to HARD_OFFLINE (ping: not OK, mysql: OK)
2010/08/13 13:20:32 INFO Removing all roles from host 'db12':
2010/08/13 13:20:32 INFO Removed role 'writer(10.17.88.140)' from host 'db12'
2010/08/13 13:20:32 INFO Orphaned role 'writer(10.17.88.140)' has been assigned to 'db13'
..
(nothing here, waiting for db12 to become online)

2010/08/13 14:16:27 FATAL Agent on host 'db12' is reachable again

db12:
--
(this one crashed, no MMM logs)

db13:

2010/08/13 14:16:24 INFO We have some new roles added or ol...

Hi,
   I like the idea of Jeon but I'd prefer the alarm code to be in the _send_command function. I'll write something about this in branch lp:~y-trudeau/mysql-mmm/NonBlocking.

Regards,

Yves

An easy way to simulate the problem. On the database host firewall the outgoing TCP PUSH packets on the agent port like this:

iptables -A OUTPUT -p tcp --sport 9989 --tcp-flags PSH PSH -j DROP

This will allow the connect to succeed, but when the agent tries to reply with data, it will block. Network-wise the behavior is the same as if the the agent/databse host ran out of memory and is not able to proceed beyond acknowledging a TCP handshake.

The monitor does have a timeout on the connect operation, but not on network I/O.

Attached is the proof-of-concept patch that fixes this problem for a likely practical scenario - a host running MySQL and the agent ran out of memory or is otherwise overloaded, and entered a state where the connection to the agent succeeds, but the actual data response is not happening. It enables timeouts on reads. This fix needs to be improved:

- take care of perpetually blocking write operation
- apply the fix to other network I/O in the code

This would require some refactoring of the code to put network I/O in wrappers.