Building images within a container fail due to size miscalculation

Bug #1702382 reported by Jamie Lennox
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
diskimage-builder
In Progress
Undecided
Jamie Lennox

Bug Description

When building Images within a container the target size calculated by `du -x` is incorrect and tiny, which results in a number of problems.

The first error message we see is: Journal size too big for filesystem

This seems to be because we are creating a file system of approximately 20mb when we intend to create something in the hundreds of mb, and so attaching a 64mb journal is a massive size difference that the tool won't create.

It appears that there is a current outstanding problem when using AUFS that the du -x flag (meaning don't cross filesystems) doesn't correctly report the files together hence our image size calculation is failing. [1]

At the point in the build where this image size calculation happens we don't have /dev /proc etc mounted so these shouldn't cause problems, however as pointed out by lifeless: "The -x is there so that if someone has mounted e.g. a apt or yum cache in one of the other hooks, we don't include it."

[1] https://patchwork.kernel.org/patch/9706629/

Changed in diskimage-builder:
assignee: nobody → Jamie Lennox (jamielennox)
status: New → In Progress
Revision history for this message
Jamie Lennox (jamielennox) wrote :

The suggestion from comment is to revert the -x removal when upstream has merged, but who knows how long that can take to get into stable kernels.

I think more likely we either:

1. Test aufs version and special case it to work around the bug.
2. Expect people to unmount volumes they don't want copied.

I think 2 is actually a more reasonable solution because the copy step appears to be executed not long after this step and it copies the image contents over regardless of filesystem and so it's possible that anything left mounted will be copied into an image that's too small. However i'm not familiar enough with the code to say that definitively.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on diskimage-builder (master)

Change abandoned by Jamie Lennox (<email address hidden>) on branch: master
Review: https://review.openstack.org/476732

Revision history for this message
Sean M. Collins (scollins) wrote :

Hi, I have run into this bug year later, we have a ConcourseCI system that is supposed to use diskimage-builder to build images and this bug seems similar to our issue

http://paste.openstack.org/show/743485/

Revision history for this message
Sean M. Collins (scollins) wrote :

Paste with the actual relevant lines

http://paste.openstack.org/show/743486/

I can provide the whole trace if required.

Revision history for this message
Manuel Torrinha (t0rrant) wrote :

Hi, this bug is still present for disk-image-create version 2.24.0

I'm running this on a gitlab-ci system, which uses docker in a gitlab runner to
build the images using diskimage-builder.

Any workaround suggestions are appreciated.

Relevant lines:

  http://paste.openstack.org/show/753451/

Revision history for this message
Manuel Torrinha (t0rrant) wrote :

I should add that this was happening with a Debian 9.9 environment, kernel 4.9.

After some search I found that this issue was related to a bug with overlayfs.

After upgrading the system to Debian 10 (unstable), kernel 4.19, this issue no longer happens.

Not sure in which version the patch was applied, probably somewhere between 4.12 and 4.15.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.