Repository corruption, index file truncated by OS file system

The bzr server repository got corrupted and nobody can pull or push changes to it. Running "bzr check" and "bzr reconcile" shows the following messages:

[11:51:11][~/trunk-r16]$ bzr merge
Merging from remembered submit location bzr+ssh://<email address hidden>/bzr/client-facebook/trunk/
bzr: ERROR: bzrlib.errors.ErrorFromSmartServer: Error received from smart server: ('ShortReadvError', 'b22a569ce92096b22330c933808dc3b1.tix', '0', '772', '60')

Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 853, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 1055, in run_bzr
    ret = run(*run_argv)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 661, in run_argv_aliases
    return self.run_direct(**all_cmd_args)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 665, in run_direct
    return self._operation.run_simple(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 122, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 156, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/builtins.py", line 3794, in run
    location, revision, remember, possible_transports, pb)
  File "/Library/Python/2.6/site-packages/bzrlib/builtins.py", line 3923, in _get_merger_from_branch
    other_revision_id, base_revision_id, other_branch, base_branch)
  File "/Library/Python/2.6/site-packages/bzrlib/merge.py", line 363, in from_revision_ids
    merger.set_other_revision(other, other_branch)
  File "/Library/Python/2.6/site-packages/bzrlib/merge.py", line 505, in set_other_revision
    self._maybe_fetch(other_branch, self.this_branch, self.other_rev_id)
  File "/Library/Python/2.6/site-packages/bzrlib/merge.py", line 522, in _maybe_fetch
    target.fetch(source, revision_id)
  File "/Library/Python/2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/branch.py", line 593, in fetch
    pb=pb)
  File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 1704, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/Library/Python/2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 3438, in fetch
    pb=pb, find_ghosts=find_ghosts)
  File "/Library/Python/2.6/site-packages/bzrlib/fetch.py", line 83, in __init__
    self.__fetch()
  File "/Library/Python/2.6/site-packages/bzrlib/fetch.py", line 109, in __fetch
    self._fetch_everything_for_search(search)
  File "/Library/Python/2.6/site-packages/bzrlib/fetch.py", line 137, in _fetch_everything_for_search
    stream, from_format, [])
  File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 4279, in insert_stream
    return self._locked_insert_stream(stream, src_format, is_resume)
  File "/Library/Python/2.6/site-packages/bzrlib/repository.py", line 4316, in _locked_insert_stream
    substream)
  File "/Library/Python/2.6/site-packages/bzrlib/knit.py", line 1599, in insert_record_stream
    for record in stream:
  File "/Library/Python/2.6/site-packages/bzrlib/versionedfile.py", line 1665, in read
    for bytes in self._bytes_iterator:
  File "/Library/Python/2.6/site-packages/bzrlib/smart/repository.py", line 571, in iter_substream_bytes
    for record in self.iter_pack_records:
  File "/Library/Python/2.6/site-packages/bzrlib/smart/repository.py", line 561, in iter_stream_decoder
    for bytes in self.byte_stream:
  File "/Library/Python/2.6/site-packages/bzrlib/smart/message.py", line 338, in read_streamed_body
    _translate_error(self._body_error_args)
  File "/Library/Python/2.6/site-packages/bzrlib/smart/message.py", line 361, in _translate_error
    raise errors.ErrorFromSmartServer(error_tuple)
ErrorFromSmartServer: Error received from smart server: ('ShortReadvError', 'b22a569ce92096b22330c933808dc3b1.tix', '0', '772', '60')

bzr 2.1.1 on python 2.6.1 (Darwin-10.4.0-i386-64bit)
arguments: ['/usr/local/bin/bzr', 'merge']
encoding: 'UTF-8', fsenc: 'utf-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /Library/Python/2.6/site-packages/bzrlib/plugins/bzrtools [2.1.0]
  email /Library/Python/2.6/site-packages/bzrlib/plugins/email [unknown]
  explorer /Library/Python/2.6/site-packages/bzrlib/plugins/explorer [1.0.1]
  extmerge /Users/dbanks/.bazaar/plugins/extmerge [unknown]
  keychain /Library/Python/2.6/site-packages/bzrlib/plugins/keychain [0.1.0]
  launchpad /Library/Python/2.6/site-packages/bzrlib/plugins/launchpad [2.1.1]
  netrc_credential_store /Library/Python/2.6/site-packages/bzrlib/plugins/netrc_credential_store [2.1.1]
  news_merge /Library/Python/2.6/site-packages/bzrlib/plugins/news_merge [2.1.1]
  qbzr /Library/Python/2.6/site-packages/bzrlib/plugins/qbzr [0.18.4]
  rebase /Library/Python/2.6/site-packages/bzrlib/plugins/rebase [0.5.5]
  svn /Library/Python/2.6/site-packages/bzrlib/plugins/svn [1.0.2]
  upload /Library/Python/2.6/site-packages/bzrlib/plugins/upload [1.0.0dev]
  xmloutput /Users/dbanks/.bazaar/plugins/xmloutput [0.8.4]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

[11:51:19][~/trunk-r16]$

build@dev-001:/bzr/client-facebook/trunk$ bzr check
Checking branch at 'file:///glass/sfw/bzr/repo/client-facebook/release/'.
Checking branch at 'file:///glass/sfw/bzr/repo/client-facebook/trunk/'.
Checking branch at 'file:///glass/sfw/bzr/repo/client-facebook/work/'.
No working tree found at specified location.
Checking repository at 'file:///glass/sfw/bzr/repo/client-facebook/'.
bzr: ERROR: bzrlib.errors.ShortReadvError: readv() read 60 bytes rather than 772 bytes at 0 for "b22a569ce92096b22330c933808dc3b1.tix"

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 3145, in run
    check_dwim(path, verbose, do_branch=branch, do_repo=repo, do_tree=tree)
  File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 456, in check_dwim
    check_repo=do_repo)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 138, in read_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2767, in check
    check_repo=check_repo)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 2771, in _check
    result.check(callback_refs)
  File "/usr/lib/python2.5/site-packages/bzrlib/check.py", line 102, in check
    self.repository._check_inventories(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1172, in _check_inventories
    self._do_check_inventories(checker, bar)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1223, in _do_check_inventories
    for record in getattr(self, kind).check(keys=keys[kind]):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1405, in get_record_stream
    ordering, include_delta_closure):
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1416, in _get_remaining_record_stream
    positions = self._get_components_positions(keys, allow_missing=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 1166, in _get_components_positions
    build_details = self._index.get_build_details(pending_components)
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 3051, in get_build_details
    for entry in entries:
  File "/usr/lib/python2.5/site-packages/bzrlib/knit.py", line 3079, in _get_entries
    for node in self._graph_index.iter_entries(keys):
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1290, in iter_entries
    for node in index.iter_entries(keys):
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 635, in iter_entries
    if self._nodes is None and len(keys) * 20 > self.key_count():
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 744, in key_count
    self._read_and_parse([_HEADER_READV])
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1170, in _read_and_parse
    for offset, data in readv_data:
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 701, in _seek_and_read
    c_offset.length, actual=len(data))
ShortReadvError: readv() read 60 bytes rather than 772 bytes at 0 for "b22a569ce92096b22330c933808dc3b1.tix"

bzr 2.0.4 on python 2.5.2 (Linux-2.6.24-26-xen-x86_64-with-debian-lenny-sid)
arguments: ['/usr/bin/bzr', 'check']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [2.0.0]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [2.0.4]
  netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [2.0.4]
  xmloutput /usr/lib/python2.5/site-packages/bzrlib/plugins/xmloutput [0.8.3]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.
build@dev-001:/bzr/client-facebook/trunk$ bzr reconcile
Reconciling branch file:///glass/sfw/bzr/repo/client-facebook/trunk/
revision_history ok.
Reconciling repository file:///glass/sfw/bzr/repo/client-facebook/
bzr: ERROR: exceptions.AssertionError: 772 59

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1597, in run
    reconcile(dir)
  File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 53, in reconcile
    reconciler.reconcile()
  File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 77, in reconcile
    self._reconcile()
  File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 84, in _reconcile
    self._reconcile_repository()
  File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 103, in _reconcile_repository
    repo_reconciler = self.repo.reconcile(thorough=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 2352, in reconcile
    reconciler.reconcile()
  File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 200, in reconcile
    self._reconcile_steps()
  File "/usr/lib/python2.5/site-packages/bzrlib/reconcile.py", line 513, in _reconcile_steps
    ".reconcile", all_revisions, self.pb)
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 2357, in _reconcile_pack
    return packer.pack(pb)
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 760, in pack
    return self._create_pack_from_packs()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 923, in _create_pack_from_packs
    self._copy_text_texts()
  File "/usr/lib/python2.5/site-packages/bzrlib/repofmt/pack_repo.py", line 1268, in _copy_text_texts
    for node in text_nodes:
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1265, in iter_all_entries
    for node in index.iter_all_entries():
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 505, in iter_all_entries
    self._buffer_all()
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 445, in _buffer_all
    _, _, _, trailers = self._parse_lines(lines, pos)
  File "/usr/lib/python2.5/site-packages/bzrlib/index.py", line 1074, in _parse_lines
    raise AssertionError("%s %s" % (self._size, pos))
AssertionError: 772 59

bzr 2.0.4 on python 2.5.2 (Linux-2.6.24-26-xen-x86_64-with-debian-lenny-sid)
arguments: ['/usr/bin/bzr', 'reconcile']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [2.0.0]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [2.0.4]
  netrc_credential_store /usr/lib/python2.5/site-packages/bzrlib/plugins/netrc_credential_store [2.0.4]
  xmloutput /usr/lib/python2.5/site-packages/bzrlib/plugins/xmloutput [0.8.3]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

ali@dev-001:/bzr/server/trunk$ bzr info -v
Repository branch (format: unnamed)
Location:
  shared repository: /glass/sfw/bzr/repo/server
  repository branch: .

Related branches:
  parent branch: /glass/sfw/bzr/repo/server/release

Format:
       control: Meta directory format 1
        branch: Branch format 7
    repository: Packs containing knits without subtree support

Branch history:
       155 revisions
        59 days old
   first revision: Fri 2010-07-16 14:19:02 -0700
  latest revision: Mon 2010-09-13 14:57:40 -0700

Repository:
      7522 revisions

Through IRC channel, people think it might be the OS file system truncating the file: expected 772 bytes, but only 59 bytes found. The OS is Ubuntu 64-bit, and has been running fine for 4 months. No system level errors or logs can be found.

Our current work around is to rollback to daily backup.

Thanks