Built-in compression does inefficient I/O

Bug #1095249 reported by Alexey Kopytov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
High
Alexey Kopytov
2.0
Fix Released
High
Alexey Kopytov
2.1
Fix Released
High
Alexey Kopytov

Bug Description

--compress does 4 unbuffered I/O operations per every 64K block of uncompressed data:

- it first writes an 8-byte compressed block header
- then it writes an 8-byte file offset
- then it writes a 4-byte checksum
- finally the compressed contents of the original 64K block is written

Each write is followed by a posix_fadvise(FADV_DONTNEED) call, which is of course very inefficient.

The fix is to buffer the output compressed stream.

Related branches

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

There is also one here:

stream_one_file(File file, xb_wstream_file_t *xbfile)
{
 uchar buf[XBSTREAM_BUFFER_SIZE];
 size_t bytes;

 posix_fadvise(file, 0, 0, POSIX_FADV_SEQUENTIAL);

 while ((bytes = my_read(file, buf, XBSTREAM_BUFFER_SIZE,
    MYF(MY_WME))) > 0) {
  if (xb_stream_write_data(xbfile, buf, bytes)) {
   msg("%s: xb_stream_write_data() failed.\n",
       my_progname);
   return 1;
  }
  posix_fadvise(file, 0, 0, POSIX_FADV_DONTNEED);

 }
......
.....

here, it is doing fadvise POSIX_FADV_SEQ which doubles the
readahead but in the loop it discards it everytime. It should
either do POSIX_FADV_DONTNEED at the end (outside the loop) or
use ranges with POSIX_FADV_DONTNEED in the loop.

Revision history for this message
Alexey Kopytov (akopytov) wrote : Re: [Bug 1095249] Re: Built-in compression does inefficient I/O

Hi Raghavendra,

On Thu, 03 Jan 2013 14:50:45 -0000, Raghavendra D Prabhu wrote:
> There is also one here:
>
>
> stream_one_file(File file, xb_wstream_file_t *xbfile)
> {
> uchar buf[XBSTREAM_BUFFER_SIZE];
> size_t bytes;
>
> posix_fadvise(file, 0, 0, POSIX_FADV_SEQUENTIAL);
>
> while ((bytes = my_read(file, buf, XBSTREAM_BUFFER_SIZE,
> MYF(MY_WME))) > 0) {
> if (xb_stream_write_data(xbfile, buf, bytes)) {
> msg("%s: xb_stream_write_data() failed.\n",
> my_progname);
> return 1;
> }
> posix_fadvise(file, 0, 0, POSIX_FADV_DONTNEED);
>
> }
> ......
> .....
>
> here, it is doing fadvise POSIX_FADV_SEQ which doubles the
> readahead but in the loop it discards it everytime. It should
> either do POSIX_FADV_DONTNEED at the end (outside the loop) or
> use ranges with POSIX_FADV_DONTNEED in the loop.
>

Agree, please report it as a separate bug, as it's not directly related
to this one. And the same is done by xtrabackup when reading the file IIRC.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

I will report it as a separate bug; as you mentioned we may need to audit other locations of its usage.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-349

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.