Mirantis OpenStack

base system installation fails due to broken deb packages' file names

Series 5.1.x
Bug #1441611

Bug #1441611 reported by Sergey Kolekonov on 2015-04-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Fix Committed	Critical	Alexei Sheplyakov	Mirantis OpenStack 5.1.1-updates
	5.1.x	Fix Committed	Critical	Alexei Sheplyakov	Mirantis OpenStack 5.1.1-updates

Bug Description

The following builds:
http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.ubuntu.bvt_2/102/
http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.ubuntu.bvt_2/103/

are broken due to provisioning timeout.

I've reverted the second environment and connected to a node using VNC.
The problem is in missing packages. The screenshot is in attachment
I've also found the following errors on nodes (10.108.10.2/var/log/docker-logs/remote/node-1.test.domain.local/anna.log):

2015-04-07T18:17:02.804744+01:00 info: http://10.108.10.2:8080/ubuntu/fuelweb/x86_64/dists/precise-updates/Release:
2015-04-07T18:17:02.805236+01:00 info: 2015-04-07 17:17:03 ERROR 404: Not Found.
2015-04-07T18:17:02.822227+01:00 info: http:///ubuntu/dists/precise-security/Release: Invalid host name.
2015-04-07T18:17:02.854193+01:00 debug: DEBUG: resolver (loop-aes-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.854193+01:00 debug: DEBUG: resolver (ufsutils-udeb): package doesn't exist (ignored)
2015-04-07T18:17:02.854218+01:00 debug: DEBUG: resolver (ufs-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.854500+01:00 debug: DEBUG: resolver (loop-aes-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.854500+01:00 debug: DEBUG: resolver (ufsutils-udeb): package doesn't exist (ignored)
2015-04-07T18:17:02.854557+01:00 debug: DEBUG: resolver (ufs-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.855473+01:00 debug: DEBUG: retrieving apt-mirror-setup 1:0.55ubuntu4

It can be a mirror problem

See original description

Tags:

Revision history for this message

Sergey Kolekonov (skolekonov) wrote on 2015-04-08:

Screenshot - 04082015 - 03:45:21 PM.png Edit (4.5 KiB, image/png)

Changed in fuel:
status:	New → Confirmed

Sergey Kolekonov (skolekonov) on 2015-04-08

description:

updated

Revision history for this message

Ivan Berezovskiy (iberezovskiy) wrote on 2015-04-08:

This issue affects only 5.1.2 staging jobs

Changed in fuel:
milestone:	none → 5.1.2
importance:	Undecided → Critical

Sergey Kolekonov (skolekonov) on 2015-04-08

summary:

- 5.1.2 staging iso is broken due to missing package bsdutils
+ 5.1.2 staging iso is broken due to missing packages during provisioning

Sergey Kolekonov (skolekonov) on 2015-04-08

Changed in fuel:
status:	Confirmed → New

Matthew Mosesohn (raytrac3r) on 2015-04-08

Changed in fuel:
assignee:	nobody → Bartlomiej Piotrowski (bpiotrowski)

Bartłomiej Piotrowski (bpiotrowski) on 2015-04-08

Changed in fuel:
assignee:	Bartlomiej Piotrowski (bpiotrowski) → Fuel OSCI Team (fuel-osci)

Roman Vyalov (r0mikiam) on 2015-04-09

Changed in fuel:
assignee:	Fuel OSCI Team (fuel-osci) → MOS Linux (mos-linux)
affects:	fuel → mos
Changed in mos:
milestone:	5.1.2 → none

Alexander Ignatov (aignatov) on 2015-04-09

Changed in mos:
milestone:	none → 5.1.2

Alexander Ignatov (aignatov) on 2015-04-10

Changed in mos:
status:	New → Confirmed

Aleksander Mogylchenko (amogylchenko) on 2015-04-10

Changed in mos:
assignee:	MOS Linux (mos-linux) → asyriy (asyriy)

Fuel Devops McRobotson (fuel-devops-robot) on 2015-04-10

Changed in mos:
milestone:	5.1.2 → 6.1
status:	Confirmed → New

Alexander Ignatov (aignatov) on 2015-04-13

Changed in mos:
status:	New → Invalid

Revision history for this message

Albert Syriy (asyriy) wrote on 2015-04-15: Re: 5.1.2 staging iso is broken due to missing packages during provisioning

The cause of the issue is lack of the following udeb packages in ISO:

bsdutils 1:2.20.1-1ubuntu3
busybox-initramfs,
diffutils
libusb
dmsetup,
iputils-ping,
libattr1,
libdermapper1.01.1,
libgcc1,
ntpdate,
passwd,
procps,
vim-common,
vim-tiny,
zlib1g

Could we check if if the the packages are on the mirror using for building ISO for 5.1.2 ?

Revision history for this message

Albert Syriy (asyriy) wrote on 2015-04-15:

This is the cause of the Ubuntu provisioning fail.

Revision history for this message

Albert Syriy (asyriy) wrote on 2015-04-15:

All the "missed" packages has the strange substitution in the package name, when the character ':' substituted by the '%3a'.
For example
----Start quotation----
Download complete and in download only mode
I: Calculating obsolete packages
.... (skipped) ....
-> Processing conffiles for bash
I: Extracting bsdutils_1%3a2.20.1-1ubuntu3.1_amd64.deb...
I: Extracting busybox-initramfs_1%3a1.18.5-1ubuntu4.1_amd64.deb...
... (skipped) ...
---------End the quotation-----
Actually should be ':'
I: Extracting bsdutils_1:2.20.1-1ubuntu3.1_amd64.deb...
I: Extracting busybox-initramfs_1:1.18.5-1ubuntu4.1_amd64.deb...

See the console log for details
http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.iso/121/consoleText

Revision history for this message

Albert Syriy (asyriy) wrote on 2015-04-21:

The status update:

The root cause of the problem is character ':' in file name of packages.
The debootstrap can't process such file name during installation and report about the packages as missing.
Looks like we have a package on our mirrors
bsdutils_1%3a2.20.1-1ubuntu3.1_amd64.deb
Which after processing (building ISO) it is saved as
bsdutils_1:2.20.1-1ubuntu3.1_amd64.deb
and this file name saved in the Packages (file describing the packages in the repository).

After fixing this issue with ":" in names (removing from the file name '1:' characters) met another issue, what investigating now.
( see as an example how the packages kept in Ubuntu repo
http://ru.archive.ubuntu.com/ubuntu/pool/main/u/util-linux/ )

Alexei Sheplyakov (asheplyakov) on 2015-04-23

summary:

- 5.1.2 staging iso is broken due to missing packages during provisioning
+ base system installation fails due to broken deb packages' file names

Revision history for this message

Alexei Sheplyakov (asheplyakov) wrote on 2015-04-23:

Actually there are two unrelated bugs:

- mirror/ubuntu/files/mkrepo.sh does not download the base packages properly, instead it copies them
  from multistrap cache. However multistrap modifies the packages' file names (that is, adds the colon
  character and epoch number) which confuses debootstrap. As a result base system installation fails
  (so the provisioning times out).
  The following patch https://review.openstack.org/176309 solves this problem.

- fuel-main builds an ISO with Linux 3.13 [0], however fuel-library insists on installing Linux 3.11 [1].
To address this issue https://review.openstack.org/152657 should be merged.

The ISO with the above mentioned patches has successfully passed BVTs [2]

[0] https://github.com/stackforge/fuel-main/blob/stable/5.1/config.mk#L103

[1] https://github.com/stackforge/fuel-library/blob/stable/5.1/deployment/puppet/cobbler/templates/preseed/ubuntu-1204.preseed.erb#L150

[2] http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.ubuntu.bvt_2/120

Revision history for this message

ruhe (ruhe) wrote on 2015-04-27:

Cleaned up the bug a little bit. Sorry for spam if there was any.

Changed in mos:
milestone:	6.1 → 5.1.2
no longer affects:	mos/6.0.x
Changed in mos:
status:	Invalid → In Progress
no longer affects:	mos
Changed in mos:
milestone:	none → 5.1.2
assignee:	nobody → Alexei Sheplyakov (asheplyakov)
importance:	Undecided → Critical
status:	New → In Progress