LXC + BTRFS: creating many containers simultaneously can cause system corruption
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
High
|
Unassigned |
Bug Description
Using lxc with a btrfs filesystem can lead to system corruption and containers which are unable to boot when many containers are created and started simultaneously.
Packages involved:
- btrfs-tools
- ubuntu-lxc/daily from http://
`lsb_release -rd`:
ubuntu@
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Here is the relevant discussion thread: http://<email address hidden>
And here is a key excerpt regarding reproducing the problem:
The general system state is something like:
N containers already running happily
Launch N+ more containers in rapid succession (in parallell, not serially).
Here is a test script which closely reflects what my application is actually doing. It slowly launches 10 containers and then uses "&" to rapidly fork an additional 10 clone/start operations. I have it doing 2 cycles of this, and running the script many times (2-6 times) eventually triggers the problem.
test.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
path=/mnt
sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1
for i in `seq 1 10`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 11 20`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
sleep 10
# Create even more.
for i in `seq 21 30`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 31 40`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
stop.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
sudo lxc-destroy -n c1;
for i in `seq 1 40`; do
echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
done
bash ./test.sh x
bash ./test.sh y
bash ./test.sh z
If it doesn't manifest at first, try starting/
x1 RUNNING - - NO
x10 RUNNING - - NO
x11 RUNNING - - NO
x12 RUNNING - - NO
x13 RUNNING - - NO
x14 RUNNING - - NO
x15 RUNNING - - NO
x16 RUNNING - - NO
x17 RUNNING - - NO
x18 RUNNING - - NO
x19 RUNNING - - NO
x2 RUNNING - - NO
x20 RUNNING - - NO
x21 RUNNING - - NO
x22 RUNNING - - NO
x23 RUNNING - - NO
x24 RUNNING - - NO
x25 RUNNING - - NO
x26 RUNNING - - NO
x27 RUNNING - - NO
x28 RUNNING - - NO
x29 RUNNING - - NO
x3 RUNNING - - NO
x30 RUNNING - - NO
x31 RUNNING - - NO
x32 RUNNING - - NO
x33 RUNNING - - NO
x34 RUNNING - - NO
x35 RUNNING - - NO
x36 RUNNING - - NO
x37 RUNNING - - NO
x38 RUNNING - - NO
x39 RUNNING - - NO
x4 RUNNING - - NO
x40 RUNNING - - NO
x5 RUNNING - - NO
x6 RUNNING - - NO
x7 RUNNING - - NO
x8 RUNNING - - NO
x9 RUNNING - - NO
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-
ProcVersionSign
Uname: Linux 3.2.0-40-virtual x86_64
AcpiTables:
AlsaDevices:
total 0
crw-rw---T 1 root audio 116, 1 Aug 14 20:40 seq
crw-rw---T 1 root audio 116, 33 Aug 14 20:40 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Mon Aug 19 18:20:19 2013
Ec2AMI: ami-d0f89fb9
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: m1.large
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MarkForUpload: True
PciMultimedia:
ProcEnviron:
TERM=xterm-
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=LABEL=
RelatedPackageV
linux-
linux-
linux-firmware 1.79.4
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog:
description: | updated |
This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:
apport-collect 1214085
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.