mount.ocfs2: join errors when node with kernel >= 2.6.37 joins with nodes with kernels < 2.6.37

my cluster.conf

node:
 ip_port = 7777
 ip_address = 192.168.79.14
 number = 0
 name = media
 cluster = ocfs2

node:
 ip_port = 7777
 ip_address = 192.168.79.77
 number = 1
 name = ibm-main
 cluster = ocfs2

node:
 ip_port = 7777
 ip_address = 192.168.79.79
 number = 2
 name = imac-lap
 cluster = ocfs2

cluster:
 node_count = 3
 name = ocfs2

my modules
# lsmod | grep ocfs
ocfs2 788119 0
quota_tree 18308 1 ocfs2
ocfs2_dlmfs 27697 1
ocfs2_stack_o2cb 13322 0
ocfs2_dlm 242947 1 ocfs2_stack_o2cb
ocfs2_nodemanager 225790 12 ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm
ocfs2_stackglue 17200 3 ocfs2,ocfs2_dlmfs,ocfs2_stack_o2cb
configfs 35105 2 ocfs2_nodemanager

On node 0 (debian 6, 1.4.4) the heartbeat parameters have the same value, which was my initial though for the "node mismatch" error, noting that ocfs2 1.6 is supposed to be fully compatible with 1.4.

The o2cb status ouput on node0 is as follows, noting the heartbeat is active because the volumes are mounted.

:~# /etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Active

Hi There,

Thank you for reporting bugs and trying to make Ubuntu better.

This actually seems like an issue with communication rather than mounting, and appears to be related to the kernel rather than ocfs2-tools package (which would made the bug belong to the kernel rather than this package).

Could you please take a look at the following thread [1] and see if these sounds familiar to you, and let us know.

For now, I'll be marking this bug report as incomplete until further information is provided.

[1]: http://<email address hidden>/msg04710.html

Thanks Andres, this is exactly it as my kernels are within the scope of this issue with exactly those symptoms. Following the thread, I noted that the upstream patch is committed and the scope is specific to the o2cb cluster stack.

http://oss.oracle.com/pipermail/ocfs2-devel/2011-April/007996.html

It is part of a set of patches for 2.6.39, but does impact 2.6.37.

http://oss.oracle.com/pipermail/ocfs2-devel/2011-April/007998.html

I successfully applied the attached patch against the current linux-image-2.6.38-8-server (.42) ubuntu source. I grabbed it from this post, and have attached it here separately.

http://oss.oracle.com/pipermail/ocfs2-devel/2011-April/007996.html

It seems like an easily reviewed merge to current 2.6.37 as it appears to simply correct an error in the code where the locking protocol version is evaluated.

I also confirmed this is pending for next bundle into upstream 2.6.39 with oracle. (i.e not in 2.6.39-rc2 changelog yet)

[ 67.429903] scsi0 : iSCSI Initiator over TCP/IP
[ 67.724378] scsi 0:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4
[ 67.724627] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 67.735449] sd 0:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[ 67.735606] sd 0:0:0:0: [sda] 7814045696 512-byte logical blocks: (4.00 TB/3.63 TiB)
[ 67.735745] sd 0:0:0:0: [sda] Write Protect is off
[ 67.735750] sd 0:0:0:0: [sda] Mode Sense: 77 00 00 08
[ 67.736025] sd 0:0:0:0: [sda] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[ 67.737612] sd 0:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[ 67.749812] sda: unknown partition table
[ 67.751647] sd 0:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[ 67.752307] sd 0:0:0:0: [sda] Attached SCSI disk
[ 72.030876] o2net: connected to node media (num 0) at 192.168.79.14:7777
[ 76.156603] OCFS2 1.5.0
[ 76.169929] o2dlm: Nodes in domain D34223F35D4749C381B259A942DE351B: 0 1
[ 76.195126] JBD: Ignoring recovery information on journal
[ 76.300303] ocfs2: Mounting device (8,0) on (node 1, slot 1) with ordered data mode.

root@ibm-main:~# uname -a
Linux ibm-main 2.6.38-8-server #42 SMP Wed Apr 13 10:32:23 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

I agree, the patch looks reasonable. Though it would be much preferable to kindly ask the patch author about the upstream submission, and when there is a commit in the upstream tree, think about SRUing it into (in fact when asking, it would be best if the upstream submission had a cc: <email address hidden> in the signed-off-by area to mark it as a fix for the 2.6.38.y stable tree). That way it won't become a potential problem in 11.10 again.

I asked this question two weeks ago, noting the Oracle team signed off on it per the thread.

Expect to see it show up anytime now in 2.6.39.

-------- Forwarded Message --------
From: Sunil Mushran <email address hidden>
Cc: Ocfs2-Devel <email address hidden>
Subject: Re: OSS o2dlm Bug reference
Date: Tue, 12 Apr 2011 17:17:55 -0700

This patch was posted a week ago and still is not in the upstream tree.
Typically Joel waits for sometime to collect patches before pushing them
upstream.

On 04/12/2011 09:14 AM, Ian B. MacDonald wrote:
> Sunil,
>
> Quick question,
>
> Is there an easy way to find this patch in upstream kernel/git? i
> browsed the changelog for 2.6.39-rcX .. lots of ocfs2 stuff, but I
> didn't see specific reference to this patch set.
>
> Is that because it is still under review?
>
> cheers,
> Ian

This bug is missing log files that will aid in dianosing the problem. From a terminal window please run:

apport-collect 756894

and then change the status of the bug back to 'New'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I'll run apport for Brad on the patched server. Unfortunately I can't bring down the OCFS2 cluster, so hopefully all the upstream data, including the Oracle patch is sufficient.

Scratch that, this headless machine barfed at apport even after oauth.

The authorization page:
 (https://launchpad.net/+authorize-token?oauth_token=-=-&allow_permission=DESKTOP_INTEGRATION)
should be opening in your browser. Use your browser to authorize
this program to access Launchpad on your behalf.

Waiting to hear from Launchpad about your decision...
ERROR: hook /usr/share/apport/package-hooks//source_linux-meta.py crashed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport/report.py", line 656, in add_hooks_info
    symb['add_info'](self, ui)
  File "/usr/share/apport/package-hooks//source_linux-meta.py", line 42, in add_info
    attach_alsa(report)
  File "/usr/lib/python2.7/dist-packages/apport/hookutils.py", line 220, in attach_alsa
    report['PciMultimedia'] = pci_devices(PCI_MULTIMEDIA)
  File "/usr/lib/python2.7/dist-packages/apport/hookutils.py", line 422, in pci_devices
    key, value = line.split(':',1)
ValueError: need more than 1 value to unpack

This is now resolved in current 11.10 packages, as well as the new 12.04 Beta

The attachment "Patch submitted by oracle to upstream 2.6.39" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]