Bug #1618429 “Curtin doesn't clean up previous MD configuration” : Bugs : curtin

Revision history for this message

Ante Karamatić (ivoks) wrote on 2016-08-30:

#1

MAAS's storage config for the node Edit (2.5 KiB, text/plain)

Revision history for this message

Ante Karamatić (ivoks) wrote on 2016-08-30:

#2

config-000.cfg Edit (2.3 KiB, text/plain)

Gábor Mészáros (gabor.meszaros) on 2016-08-30

tags:

added: 4010

Revision history for this message

Ante Karamatić (ivoks) wrote on 2016-08-30:

#3

I forgot to mention, this is curtin in 14.04. I'd assume that's 0.1.0~bzr399-0ubuntu1~16.04.1.

Revision history for this message

Ryan Harper (raharper) wrote on 2016-08-30: Re: [Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

#4

Hi Ante,

Can you attach the install.log reported back to maas? It should be on the
node details page.

https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b

On Tue, Aug 30, 2016 at 7:02 AM, Ante Karamatić <
<email address hidden>> wrote:

> I forgot to mention, this is curtin in 14.04. I'd assume that's
> 0.1.0~bzr399-0ubuntu1~16.04.1.
>
> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

Revision history for this message

Ante Karamatić (ivoks) wrote on 2016-08-30:

#5

I don't have from that specific machine, because I had to move on. But
error was pretty much the same as this one (maybe md3 instead of md4):

Error: /dev/sda: unrecognised disk labelAn error occured handling
'sda': ProcessExecutionError - Unexpected error while running
command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
-Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
enough to start the array.\n'Unexpected error while running
command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
-Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
enough to start the array.\n'Installation failed with exception:
Unexpected error while running command.Command: ['curtin',
'block-meta', 'custom']Exit code: 3Reason: -Stdout: "Error: /dev/sda:
unrecognised disk label\nAn error occured handling 'sda':
ProcessExecutionError - Unexpected error while running
command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit code:
3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled from 3
drives - not enough to start the array.\\n'\nUnexpected error while
running command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit
code: 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled
from 3 drives - not enough to start the array.\\n'\n"Stderr: ''

On Tue, Aug 30, 2016 at 3:40 PM Ryan Harper <email address hidden>
wrote:

> Hi Ante,
>
> Can you attach the install.log reported back to maas? It should be on the
> node details page.
>
> https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b
>
> On Tue, Aug 30, 2016 at 7:02 AM, Ante Karamatić <
> <email address hidden>> wrote:
>
> > I forgot to mention, this is curtin in 14.04. I'd assume that's
> > 0.1.0~bzr399-0ubuntu1~16.04.1.
> >
> > --
> > You received this bug notification because you are subscribed to curtin.
> > Matching subscriptions: curtin-bugs-all
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> > Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>
--
Ante Karamatić
<email address hidden>
Canonical

I don't have from that specific machine, because I had to move on. But
error was pretty much the same as this one (maybe md3 instead of md4):

Error: /dev/sda: unrecognised disk labelAn error occured handling
'sda': ProcessExecutionError - Unexpected error while running
command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
-Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
enough to start the array.\n'Unexpected error while running
command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
-Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
enough to start the array.\n'Installation failed with exception:
Unexpected error while running command.Command: ['curtin',
'block-meta', 'custom']Exit code: 3Reason: -Stdout: "Error: /dev/sda:
unrecognised disk label\nAn error occured handling 'sda':
ProcessExecutionError - Unexpected error while running
command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit code:
3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled from 3
drives - not enough to start the array.\\n'\nUnexpected error while
running command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit
code: 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled
from 3 drives - not enough to start the array.\\n'\n"Stderr: ''

On Tue, Aug 30, 2016 at 3:40 PM Ryan Harper <1618429@bugs.launchpad.net>
wrote:

> Hi Ante,
>
> Can you attach the install.log reported back to maas?  It should be on the
> node details page.
>
> https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b
>
> On Tue, Aug 30, 2016 at 7:02 AM, Ante Karamatić <
> ante.karamatic@canonical.com> wrote:
>
> > I forgot to mention, this is curtin in 14.04. I'd assume that's
> > 0.1.0~bzr399-0ubuntu1~16.04.1.
> >
> > --
> > You received this bug notification because you are subscribed to curtin.
> > Matching subscriptions: curtin-bugs-all
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> >   Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
>   Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>
-- 
Ante Karamatić
ante.karamatic@canonical.com
Canonical

Revision history for this message

Ryan Harper (raharper) wrote on 2016-08-30:

#6

OK, if you do run into it again or reproduce it, the entire log is
important, not just the stack trace.

On Tue, Aug 30, 2016 at 8:54 AM, Ante Karamatić <
<email address hidden>> wrote:

> I don't have from that specific machine, because I had to move on. But
> error was pretty much the same as this one (maybe md3 instead of md4):
>
> Error: /dev/sda: unrecognised disk labelAn error occured handling
> 'sda': ProcessExecutionError - Unexpected error while running
> command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> enough to start the array.\n'Unexpected error while running
> command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> enough to start the array.\n'Installation failed with exception:
> Unexpected error while running command.Command: ['curtin',
> 'block-meta', 'custom']Exit code: 3Reason: -Stdout: "Error: /dev/sda:
> unrecognised disk label\nAn error occured handling 'sda':
> ProcessExecutionError - Unexpected error while running
> command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit code:
> 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled from 3
> drives - not enough to start the array.\\n'\nUnexpected error while
> running command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit
> code: 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled
> from 3 drives - not enough to start the array.\\n'\n"Stderr: ''
>
>
> On Tue, Aug 30, 2016 at 3:40 PM Ryan Harper <email address hidden>
> wrote:
>
> > Hi Ante,
> >
> > Can you attach the install.log reported back to maas? It should be on
> the
> > node details page.
> >
> > https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b
> >
> > On Tue, Aug 30, 2016 at 7:02 AM, Ante Karamatić <
> > <email address hidden>> wrote:
> >
> > > I forgot to mention, this is curtin in 14.04. I'd assume that's
> > > 0.1.0~bzr399-0ubuntu1~16.04.1.
> > >
> > > --
> > > You received this bug notification because you are subscribed to
> curtin.
> > > Matching subscriptions: curtin-bugs-all
> > > https://bugs.launchpad.net/bugs/1618429
> > >
> > > Title:
> > > Curtin doesn't clean up previous MD configuration
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > >
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> > Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
> --
> Ante Karamatić
> <email address hidden>
> Canonical
>
> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

OK, if you do run into it again or reproduce it, the entire log is
important, not just the stack trace.

On Tue, Aug 30, 2016 at 8:54 AM, Ante Karamatić <
ante.karamatic@canonical.com> wrote:

> I don't have from that specific machine, because I had to move on. But
> error was pretty much the same as this one (maybe md3 instead of md4):
>
> Error: /dev/sda: unrecognised disk labelAn error occured handling
> 'sda': ProcessExecutionError - Unexpected error while running
> command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> enough to start the array.\n'Unexpected error while running
> command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> enough to start the array.\n'Installation failed with exception:
> Unexpected error while running command.Command: ['curtin',
> 'block-meta', 'custom']Exit code: 3Reason: -Stdout: "Error: /dev/sda:
> unrecognised disk label\nAn error occured handling 'sda':
> ProcessExecutionError - Unexpected error while running
> command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit code:
> 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled from 3
> drives - not enough to start the array.\\n'\nUnexpected error while
> running command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit
> code: 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled
> from 3 drives - not enough to start the array.\\n'\n"Stderr: ''
>
>
> On Tue, Aug 30, 2016 at 3:40 PM Ryan Harper <1618429@bugs.launchpad.net>
> wrote:
>
> > Hi Ante,
> >
> > Can you attach the install.log reported back to maas?  It should be on
> the
> > node details page.
> >
> > https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b
> >
> > On Tue, Aug 30, 2016 at 7:02 AM, Ante Karamatić <
> > ante.karamatic@canonical.com> wrote:
> >
> > > I forgot to mention, this is curtin in 14.04. I'd assume that's
> > > 0.1.0~bzr399-0ubuntu1~16.04.1.
> > >
> > > --
> > > You received this bug notification because you are subscribed to
> curtin.
> > > Matching subscriptions: curtin-bugs-all
> > > https://bugs.launchpad.net/bugs/1618429
> > >
> > > Title:
> > >   Curtin doesn't clean up previous MD configuration
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > >
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> >   Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
> --
> Ante Karamatić
> ante.karamatic@canonical.com
> Canonical
>
> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
>   Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

Revision history for this message

Ante Karamatić (ivoks) wrote on 2016-08-30:

#7

Download full text (3.8 KiB)

That's all there was in installation log. Installation couldn't go further
without the disk, so that was all.

On Tue, Aug 30, 2016 at 4:10 PM Ryan Harper <email address hidden>
wrote:

> OK, if you do run into it again or reproduce it, the entire log is
> important, not just the stack trace.
>
> On Tue, Aug 30, 2016 at 8:54 AM, Ante Karamatić <
> <email address hidden>> wrote:
>
> > I don't have from that specific machine, because I had to move on. But
> > error was pretty much the same as this one (maybe md3 instead of md4):
> >
> > Error: /dev/sda: unrecognised disk labelAn error occured handling
> > 'sda': ProcessExecutionError - Unexpected error while running
> > command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> > -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> > enough to start the array.\n'Unexpected error while running
> > command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> > -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> > enough to start the array.\n'Installation failed with exception:
> > Unexpected error while running command.Command: ['curtin',
> > 'block-meta', 'custom']Exit code: 3Reason: -Stdout: "Error: /dev/sda:
> > unrecognised disk label\nAn error occured handling 'sda':
> > ProcessExecutionError - Unexpected error while running
> > command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit code:
> > 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled from 3
> > drives - not enough to start the array.\\n'\nUnexpected error while
> > running command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit
> > code: 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled
> > from 3 drives - not enough to start the array.\\n'\n"Stderr: ''
> >
> >
> > On Tue, Aug 30, 2016 at 3:40 PM Ryan Harper <email address hidden>
> > wrote:
> >
> > > Hi Ante,
> > >
> > > Can you attach the install.log reported back to maas? It should be on
> > the
> > > node details page.
> > >
> > > https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b
> > >
> > > On Tue, Aug 30, 2016 at 7:02 AM, Ante Karamatić <
> > > <email address hidden>> wrote:
> > >
> > > > I forgot to mention, this is curtin in 14.04. I'd assume that's
> > > > 0.1.0~bzr399-0ubuntu1~16.04.1.
> > > >
> > > > --
> > > > You received this bug notification because you are subscribed to
> > curtin.
> > > > Matching subscriptions: curtin-bugs-all
> > > > https://bugs.launchpad.net/bugs/1618429
> > > >
> > > > Title:
> > > > Curtin doesn't clean up previous MD configuration
> > > >
> > > > To manage notifications about this bug go to:
> > > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > > >
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1618429
> > >
> > > Title:
> > > Curtin doesn't clean up previous MD configuration
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > >
> > --
> > Ante Karamatić
> > <email address hidden>
> > Canonical
> >
> > --
> > ...

That's all there was in installation log. Installation couldn't go further
without the disk, so that was all.

On Tue, Aug 30, 2016 at 4:10 PM Ryan Harper <1618429@bugs.launchpad.net>
wrote:

> OK, if you do run into it again or reproduce it, the entire log is
> important, not just the stack trace.
>
> On Tue, Aug 30, 2016 at 8:54 AM, Ante Karamatić <
> ante.karamatic@canonical.com> wrote:
>
> > I don't have from that specific machine, because I had to move on. But
> > error was pretty much the same as this one (maybe md3 instead of md4):
> >
> > Error: /dev/sda: unrecognised disk labelAn error occured handling
> > 'sda': ProcessExecutionError - Unexpected error while running
> > command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> > -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> > enough to start the array.\n'Unexpected error while running
> > command.Command: ['mdadm', '--assemble', '--scan']Exit code: 3Reason:
> > -Stdout: ''Stderr: u'mdadm: /dev/md/4 assembled from 3 drives - not
> > enough to start the array.\n'Installation failed with exception:
> > Unexpected error while running command.Command: ['curtin',
> > 'block-meta', 'custom']Exit code: 3Reason: -Stdout: "Error: /dev/sda:
> > unrecognised disk label\nAn error occured handling 'sda':
> > ProcessExecutionError - Unexpected error while running
> > command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit code:
> > 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled from 3
> > drives - not enough to start the array.\\n'\nUnexpected error while
> > running command.\nCommand: ['mdadm', '--assemble', '--scan']\nExit
> > code: 3\nReason: -\nStdout: ''\nStderr: u'mdadm: /dev/md/4 assembled
> > from 3 drives - not enough to start the array.\\n'\n"Stderr: ''
> >
> >
> > On Tue, Aug 30, 2016 at 3:40 PM Ryan Harper <1618429@bugs.launchpad.net>
> > wrote:
> >
> > > Hi Ante,
> > >
> > > Can you attach the install.log reported back to maas?  It should be on
> > the
> > > node details page.
> > >
> > > https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b
> > >
> > > On Tue, Aug 30, 2016 at 7:02 AM, Ante Karamatić <
> > > ante.karamatic@canonical.com> wrote:
> > >
> > > > I forgot to mention, this is curtin in 14.04. I'd assume that's
> > > > 0.1.0~bzr399-0ubuntu1~16.04.1.
> > > >
> > > > --
> > > > You received this bug notification because you are subscribed to
> > curtin.
> > > > Matching subscriptions: curtin-bugs-all
> > > > https://bugs.launchpad.net/bugs/1618429
> > > >
> > > > Title:
> > > >   Curtin doesn't clean up previous MD configuration
> > > >
> > > > To manage notifications about this bug go to:
> > > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > > >
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1618429
> > >
> > > Title:
> > >   Curtin doesn't clean up previous MD configuration
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > >
> > --
> > Ante Karamatić
> > ante.karamatic@canonical.com
> > Canonical
> >
> > --
> > You received this bug notification because you are subscribed to curtin.
> > Matching subscriptions: curtin-bugs-all
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> >   Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
>   Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>
-- 
Ante Karamatić
ante.karamatic@canonical.com
Canonical

Revision history for this message

Ryan Harper (raharper) wrote on 2016-08-30:

#8

The flow of execution for this bug is:

curtin is instructed to wipe the superblock of the devices in the storage config.
For each disk, we need to attempt to see if it may be part of mdadm array which allows curtin to determine how to wipe mdadm metadata. In this process, curtin invokes the mdadm --assemble --scan mode to discover the possible arrays. The current code that's been tested expected return codes of 0, 1 or 2. However, in your case, it's returned 3; this is an unexpected output.

I've not been able to recreate the mdadm --assemble --scan return code of 3; the mdadm code does not easily reveal why it returns 3 in the above case versus the consistent return code of 1 when I recreate missing elements of the array on trusty.

root@ubuntu:/curtin# mdadm --assemble --scan -vv
mdadm: looking for devices for /dev/md/0
mdadm: no RAID superblock on /dev/sdi
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda2
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: cannot open device /dev/sr0: No medium found
mdadm: no RAID superblock on /dev/vda
mdadm: /dev/sdh is identified as a member of /dev/md/0, slot 6.
mdadm: /dev/sdg is identified as a member of /dev/md/0, slot 5.
mdadm: /dev/sdf is identified as a member of /dev/md/0, slot 4.
mdadm: no uptodate device for slot 0 of /dev/md/0
mdadm: no uptodate device for slot 1 of /dev/md/0
mdadm: no uptodate device for slot 2 of /dev/md/0
mdadm: no uptodate device for slot 3 of /dev/md/0
mdadm: added /dev/sdg to /dev/md/0 as 5
mdadm: added /dev/sdh to /dev/md/0 as 6
mdadm: no uptodate device for slot 7 of /dev/md/0
mdadm: added /dev/sdf to /dev/md/0 as 4
mdadm: /dev/md/0 assembled from 3 drives - not enough to start the array.
root@ubuntu:/curtin# echo $?
1

That said, it's unlikely that we actually care about any assemble errors doing the portion of the code where we're attempting to wipe a disk of any previous metadata. We're currently investigating a case where failure to assemble an array prevented new arrays from being constructed (lvm for example refuses to use devices with mdadm metadata present). If we can determine that assemble is required, we'll update the curtin mdadm code to allow us to catch and ignore errors during the wipe case as needed.

The flow of execution for this bug is:

curtin is instructed to wipe the superblock of the devices in the storage config.
For each disk, we need to attempt to see if it may be part of mdadm array which allows curtin to determine how to wipe mdadm metadata.  In this process, curtin invokes the mdadm --assemble --scan mode to discover the possible arrays.  The current code that's been tested expected return codes of 0, 1 or 2.  However, in your case, it's returned 3; this is an unexpected output.

I've not been able to recreate the mdadm --assemble --scan return code of 3;  the mdadm code does not easily reveal why it returns 3 in the above case versus the consistent return code of 1 when I recreate missing elements of the array on trusty.

root@ubuntu:/curtin# mdadm --assemble --scan -vv
mdadm: looking for devices for /dev/md/0
mdadm: no RAID superblock on /dev/sdi
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda2
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: cannot open device /dev/sr0: No medium found
mdadm: no RAID superblock on /dev/vda
mdadm: /dev/sdh is identified as a member of /dev/md/0, slot 6.
mdadm: /dev/sdg is identified as a member of /dev/md/0, slot 5.
mdadm: /dev/sdf is identified as a member of /dev/md/0, slot 4.
mdadm: no uptodate device for slot 0 of /dev/md/0
mdadm: no uptodate device for slot 1 of /dev/md/0
mdadm: no uptodate device for slot 2 of /dev/md/0
mdadm: no uptodate device for slot 3 of /dev/md/0
mdadm: added /dev/sdg to /dev/md/0 as 5
mdadm: added /dev/sdh to /dev/md/0 as 6
mdadm: no uptodate device for slot 7 of /dev/md/0
mdadm: added /dev/sdf to /dev/md/0 as 4
mdadm: /dev/md/0 assembled from 3 drives - not enough to start the array.
root@ubuntu:/curtin# echo $?
1

That said, it's unlikely that we actually care about any assemble errors doing the portion of the code where we're attempting to wipe a disk of any previous metadata.  We're currently investigating a case where failure to assemble an array prevented new arrays from being constructed (lvm for example refuses to use devices with mdadm metadata present).  If we can determine that assemble is required, we'll update the curtin mdadm code to allow us to catch and ignore errors during the wipe case as needed.

Changed in curtin:
importance:	Undecided → High
status:	New → Confirmed

Andres Rodriguez (andreserl) on 2016-09-14

tags:

added: maas

Joey Stanford (joey) on 2016-09-14

tags:	added: bootstack
tags:	added: canonical-bootstack removed: bootstack

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-15:

#9

This affects some other deployment. All details, including logs, have been provided at https://bugs.launchpad.net/maas/+bug/1623481

Revision history for this message

Ryan Harper (raharper) wrote on 2016-09-15:

#10

A build should show up in the curtin-daily ppa in a bit:

https://code.launchpad.net/~curtin-dev/+archive/ubuntu/daily/

Changed in curtin:
status:	Confirmed → Fix Committed

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-16:

#11

Download full text (16.8 KiB)

Ryan,

After installing 'curtin' package from 'ppa:curtin-dev/daily' PPA on the MaaS node:

bootstack@os-maas-1:~$ sudo add-apt-repository ppa:curtin-dev/daily

More info: https://launchpad.net/~curtin-dev/+archive/ubuntu/daily
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpbr4t6tkh/secring.gpg' created
gpg: keyring `/tmp/tmpbr4t6tkh/pubring.gpg' created
gpg: requesting key 0165013E from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpbr4t6tkh/trustdb.gpg: trustdb created
gpg: key 0165013E: public key "Launchpad PPA for curtin developers" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
bootstack@os-maas-1:~$ sudo apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Get:1 http://us.archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Hit http://us.archive.ubuntu.com trusty-backports InRelease
Hit http://us.archive.ubuntu.com trusty Release.gpg
Get:2 http://us.archive.ubuntu.com trusty-updates/main Sources [381 kB]
Get:3 http://us.archive.ubuntu.com trusty-updates/restricted Sources [5,360 B]
Get:4 http://us.archive.ubuntu.com trusty-updates/universe Sources [164 kB]
Get:5 http://us.archive.ubuntu.com trusty-updates/multiverse Sources [7,126 B]
Hit http://security.ubuntu.com trusty-security InRelease
Get:6 http://ppa.launchpad.net trusty InRelease [16.0 kB]
Hit http://security.ubuntu.com trusty-security/main Sources
Hit http://security.ubuntu.com trusty-security/restricted Sources
Get:7 http://us.archive.ubuntu.com trusty-updates/main amd64 Packages [893 kB]
Hit http://ppa.launchpad.net trusty InRelease
Get:8 http://us.archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.9 kB]
Get:9 http://us.archive.ubuntu.com trusty-updates/universe amd64 Packages [374 kB]
Get:10 http://us.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.8 kB]
Hit http://security.ubuntu.com trusty-security/universe Sources
Get:11 http://us.archive.ubuntu.com trusty-updates/main i386 Packages [855 kB]
Hit http://ppa.launchpad.net trusty InRelease
Get:12 http://us.archive.ubuntu.com trusty-updates/restricted i386 Packages [15.6 kB]
Get:13 http://us.archive.ubuntu.com trusty-updates/universe i386 Packages [375 kB]
Get:14 http://us.archive.ubuntu.com trusty-updates/multiverse i386 Packages [15.2 kB]
Hit http://security.ubuntu.com trusty-security/multiverse Sources
Hit http://us.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://us.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://us.archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://us.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://us.archive.ubuntu.com trusty Release
Get:15 http://ppa.launchpad.net trusty/main amd64 Packages [1,109 B]
Hit http://us.arc...

Ryan,

After installing 'curtin' package from 'ppa:curtin-dev/daily' PPA on the MaaS node:

bootstack@os-maas-1:~$ sudo add-apt-repository ppa:curtin-dev/daily
 
 More info: https://launchpad.net/~curtin-dev/+archive/ubuntu/daily
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpbr4t6tkh/secring.gpg' created
gpg: keyring `/tmp/tmpbr4t6tkh/pubring.gpg' created
gpg: requesting key 0165013E from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpbr4t6tkh/trustdb.gpg: trustdb created
gpg: key 0165013E: public key "Launchpad PPA for curtin developers" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
bootstack@os-maas-1:~$ sudo apt-get update
Ign http://us.archive.ubuntu.com trusty InRelease
Get:1 http://us.archive.ubuntu.com trusty-updates InRelease [65.9 kB]          
Hit http://us.archive.ubuntu.com trusty-backports InRelease                    
Hit http://us.archive.ubuntu.com trusty Release.gpg                            
Get:2 http://us.archive.ubuntu.com trusty-updates/main Sources [381 kB]        
Get:3 http://us.archive.ubuntu.com trusty-updates/restricted Sources [5,360 B] 
Get:4 http://us.archive.ubuntu.com trusty-updates/universe Sources [164 kB]    
Get:5 http://us.archive.ubuntu.com trusty-updates/multiverse Sources [7,126 B] 
Hit http://security.ubuntu.com trusty-security InRelease                       
Get:6 http://ppa.launchpad.net trusty InRelease [16.0 kB]                      
Hit http://security.ubuntu.com trusty-security/main Sources                    
Hit http://security.ubuntu.com trusty-security/restricted Sources              
Get:7 http://us.archive.ubuntu.com trusty-updates/main amd64 Packages [893 kB] 
Hit http://ppa.launchpad.net trusty InRelease                                 
Get:8 http://us.archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.9 kB]
Get:9 http://us.archive.ubuntu.com trusty-updates/universe amd64 Packages [374 kB]
Get:10 http://us.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.8 kB]
Hit http://security.ubuntu.com trusty-security/universe Sources                
Get:11 http://us.archive.ubuntu.com trusty-updates/main i386 Packages [855 kB] 
Hit http://ppa.launchpad.net trusty InRelease                                  
Get:12 http://us.archive.ubuntu.com trusty-updates/restricted i386 Packages [15.6 kB]
Get:13 http://us.archive.ubuntu.com trusty-updates/universe i386 Packages [375 kB]
Get:14 http://us.archive.ubuntu.com trusty-updates/multiverse i386 Packages [15.2 kB]
Hit http://security.ubuntu.com trusty-security/multiverse Sources              
Hit http://us.archive.ubuntu.com trusty-updates/main Translation-en            
Hit http://us.archive.ubuntu.com trusty-updates/multiverse Translation-en      
Hit http://us.archive.ubuntu.com trusty-updates/restricted Translation-en      
Hit http://security.ubuntu.com trusty-security/main amd64 Packages             
Hit http://us.archive.ubuntu.com trusty-updates/universe Translation-en        
Hit http://us.archive.ubuntu.com trusty Release                                
Get:15 http://ppa.launchpad.net trusty/main amd64 Packages [1,109 B]           
Hit http://us.archive.ubuntu.com trusty-backports/main Sources                 
Hit http://us.archive.ubuntu.com trusty-backports/restricted Sources           
Hit http://us.archive.ubuntu.com trusty-backports/universe Sources             
Hit http://us.archive.ubuntu.com trusty-backports/multiverse Sources           
Hit http://us.archive.ubuntu.com trusty-backports/main amd64 Packages          
Hit http://us.archive.ubuntu.com trusty-backports/restricted amd64 Packages    
Hit http://us.archive.ubuntu.com trusty-backports/universe amd64 Packages      
Hit http://us.archive.ubuntu.com trusty-backports/multiverse amd64 Packages    
Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages       
Hit http://us.archive.ubuntu.com trusty-backports/main i386 Packages           
Hit http://us.archive.ubuntu.com trusty-backports/restricted i386 Packages     
Hit http://us.archive.ubuntu.com trusty-backports/universe i386 Packages       
Get:16 http://ppa.launchpad.net trusty/main i386 Packages [1,109 B]            
Hit http://us.archive.ubuntu.com trusty-backports/multiverse i386 Packages     
Hit http://us.archive.ubuntu.com trusty-backports/main Translation-en          
Hit http://us.archive.ubuntu.com trusty-backports/multiverse Translation-en    
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages         
Hit http://us.archive.ubuntu.com trusty-backports/restricted Translation-en    
Hit http://us.archive.ubuntu.com trusty-backports/universe Translation-en      
Hit http://us.archive.ubuntu.com trusty/main Sources                           
Get:17 http://ppa.launchpad.net trusty/main Translation-en [387 B]             
Hit http://us.archive.ubuntu.com trusty/restricted Sources                     
Hit http://us.archive.ubuntu.com trusty/universe Sources                       
Hit http://us.archive.ubuntu.com trusty/multiverse Sources                     
Hit http://us.archive.ubuntu.com trusty/main amd64 Packages                    
Hit http://us.archive.ubuntu.com trusty/restricted amd64 Packages              
Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages       
Hit http://us.archive.ubuntu.com trusty/universe amd64 Packages                
Hit http://us.archive.ubuntu.com trusty/multiverse amd64 Packages              
Hit http://us.archive.ubuntu.com trusty/main i386 Packages                     
Hit http://us.archive.ubuntu.com trusty/restricted i386 Packages               
Hit http://us.archive.ubuntu.com trusty/universe i386 Packages                 
Hit http://ppa.launchpad.net trusty/main amd64 Packages                        
Hit http://us.archive.ubuntu.com trusty/multiverse i386 Packages               
Hit http://us.archive.ubuntu.com trusty/main Translation-en                    
Hit http://us.archive.ubuntu.com trusty/multiverse Translation-en              
Hit http://security.ubuntu.com trusty-security/main i386 Packages              
Hit http://ppa.launchpad.net trusty/main i386 Packages                         
Hit http://us.archive.ubuntu.com trusty/restricted Translation-en              
Hit http://us.archive.ubuntu.com trusty/universe Translation-en                
Hit http://security.ubuntu.com trusty-security/restricted i386 Packages        
Hit http://ppa.launchpad.net trusty/main Translation-en                        
Hit http://security.ubuntu.com trusty-security/universe i386 Packages          
Ign http://us.archive.ubuntu.com trusty/main Translation-en_US                 
Hit http://ppa.launchpad.net trusty/main amd64 Packages                        
Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US           
Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US           
Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US             
Hit http://security.ubuntu.com trusty-security/multiverse i386 Packages        
Hit http://ppa.launchpad.net trusty/main i386 Packages            
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://ppa.launchpad.net trusty/main Translation-en           
Hit http://security.ubuntu.com trusty-security/multiverse Translation-en
Hit http://security.ubuntu.com trusty-security/restricted Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Fetched 3,200 kB in 3s (1,027 kB/s)           
Reading package lists... Done
bootstack@os-maas-1:~$ sudo apt install curtin
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  bcache-tools btrfs-tools curtin-common gdisk libdevmapper-event1.02.1
  libicu52 liblzo2-2 lvm2 mdadm postfix python-curtin python3-crypto
  python3-curtin python3-oauthlib python3-yaml watershed xfsprogs
Suggested packages:
  thin-provisioning-tools procmail postfix-mysql postfix-pgsql postfix-ldap
  postfix-pcre sasl2-bin dovecot-common postfix-cdb mail-reader postfix-doc
  python3-crypto-dbg python-crypto-doc xfsdump acl attr quota
Recommended packages:
  default-mta mail-transport-agent
The following NEW packages will be installed:
  bcache-tools btrfs-tools curtin gdisk libdevmapper-event1.02.1 libicu52
  liblzo2-2 lvm2 mdadm postfix python3-crypto python3-curtin python3-oauthlib
  python3-yaml watershed xfsprogs
The following packages will be upgraded:
  curtin-common python-curtin
2 upgraded, 16 newly installed, 0 to remove and 10 not upgraded.
Need to get 10.3 MB of archives.
After this operation, 43.2 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/main libdevmapper-event1.02.1 amd64 2:1.02.77-6ubuntu2 [10.8 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main libicu52 amd64 52.1-3ubuntu0.4 [6,752 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main liblzo2-2 amd64 2.06-1.2ubuntu1.1 [46.1 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main bcache-tools amd64 1.0.7-1~14.04.1 [17.4 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main btrfs-tools amd64 3.12-1ubuntu0.1 [334 kB]
Get:6 http://ppa.launchpad.net/curtin-dev/daily/ubuntu/ trusty/main python-curtin all 0.1.0~bzr419-0ubuntu1~ubuntu14.04.1 [76.7 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu/ trusty/main watershed amd64 7 [11.4 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu/ trusty/main lvm2 amd64 2.02.98-6ubuntu2 [470 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main mdadm amd64 3.2.5-5ubuntu4.3 [362 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main postfix amd64 2.11.0-1ubuntu1 [1,084 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu/ trusty/main python3-crypto amd64 2.6.1-4build1 [239 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu/ trusty/main python3-oauthlib all 0.6.1-1 [56.7 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main python3-yaml amd64 3.10-4ubuntu0.1 [92.7 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu/ trusty/main xfsprogs amd64 3.1.9ubuntu2 [508 kB]
Get:15 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main gdisk amd64 0.8.8-1ubuntu0.1 [185 kB]
Get:16 http://ppa.launchpad.net/curtin-dev/daily/ubuntu/ trusty/main curtin-common all 0.1.0~bzr419-0ubuntu1~ubuntu14.04.1 [10.8 kB]
Get:17 http://ppa.launchpad.net/curtin-dev/daily/ubuntu/ trusty/main python3-curtin all 0.1.0~bzr419-0ubuntu1~ubuntu14.04.1 [76.8 kB]
Get:18 http://ppa.launchpad.net/curtin-dev/daily/ubuntu/ trusty/main curtin all 0.1.0~bzr419-0ubuntu1~ubuntu14.04.1 [2,520 B]
Fetched 10.3 MB in 0s (14.0 MB/s)    
Preconfiguring packages ...
Selecting previously unselected package libdevmapper-event1.02.1:amd64.
(Reading database ... 109516 files and directories currently installed.)
Preparing to unpack .../libdevmapper-event1.02.1_2%3a1.02.77-6ubuntu2_amd64.deb ...
Unpacking libdevmapper-event1.02.1:amd64 (2:1.02.77-6ubuntu2) ...
Selecting previously unselected package libicu52:amd64.
Preparing to unpack .../libicu52_52.1-3ubuntu0.4_amd64.deb ...
Unpacking libicu52:amd64 (52.1-3ubuntu0.4) ...
Selecting previously unselected package liblzo2-2:amd64.
Preparing to unpack .../liblzo2-2_2.06-1.2ubuntu1.1_amd64.deb ...
Unpacking liblzo2-2:amd64 (2.06-1.2ubuntu1.1) ...
Selecting previously unselected package bcache-tools.
Preparing to unpack .../bcache-tools_1.0.7-1~14.04.1_amd64.deb ...
Unpacking bcache-tools (1.0.7-1~14.04.1) ...
Selecting previously unselected package btrfs-tools.
Preparing to unpack .../btrfs-tools_3.12-1ubuntu0.1_amd64.deb ...
Unpacking btrfs-tools (3.12-1ubuntu0.1) ...
Selecting previously unselected package watershed.
Preparing to unpack .../archives/watershed_7_amd64.deb ...
Unpacking watershed (7) ...
Selecting previously unselected package lvm2.
Preparing to unpack .../lvm2_2.02.98-6ubuntu2_amd64.deb ...
Unpacking lvm2 (2.02.98-6ubuntu2) ...
Selecting previously unselected package mdadm.
Preparing to unpack .../mdadm_3.2.5-5ubuntu4.3_amd64.deb ...
Unpacking mdadm (3.2.5-5ubuntu4.3) ...
Selecting previously unselected package postfix.
Preparing to unpack .../postfix_2.11.0-1ubuntu1_amd64.deb ...
Unpacking postfix (2.11.0-1ubuntu1) ...
Selecting previously unselected package python3-crypto.
Preparing to unpack .../python3-crypto_2.6.1-4build1_amd64.deb ...
Unpacking python3-crypto (2.6.1-4build1) ...
Selecting previously unselected package python3-oauthlib.
Preparing to unpack .../python3-oauthlib_0.6.1-1_all.deb ...
Unpacking python3-oauthlib (0.6.1-1) ...
Selecting previously unselected package python3-yaml.
Preparing to unpack .../python3-yaml_3.10-4ubuntu0.1_amd64.deb ...
Unpacking python3-yaml (3.10-4ubuntu0.1) ...
Selecting previously unselected package xfsprogs.
Preparing to unpack .../xfsprogs_3.1.9ubuntu2_amd64.deb ...
Unpacking xfsprogs (3.1.9ubuntu2) ...
Selecting previously unselected package gdisk.
Preparing to unpack .../gdisk_0.8.8-1ubuntu0.1_amd64.deb ...
Unpacking gdisk (0.8.8-1ubuntu0.1) ...
Preparing to unpack .../python-curtin_0.1.0~bzr419-0ubuntu1~ubuntu14.04.1_all.deb ...
Unpacking python-curtin (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) over (0.1.0~bzr399-0ubuntu1~14.04.1) ...
Preparing to unpack .../curtin-common_0.1.0~bzr419-0ubuntu1~ubuntu14.04.1_all.deb ...
Unpacking curtin-common (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) over (0.1.0~bzr399-0ubuntu1~14.04.1) ...
Selecting previously unselected package python3-curtin.
Preparing to unpack .../python3-curtin_0.1.0~bzr419-0ubuntu1~ubuntu14.04.1_all.deb ...
Unpacking python3-curtin (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) ...
Selecting previously unselected package curtin.
Preparing to unpack .../curtin_0.1.0~bzr419-0ubuntu1~ubuntu14.04.1_all.deb ...
Unpacking curtin (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for ufw (0.34~rc-0ubuntu2) ...
Setting up libdevmapper-event1.02.1:amd64 (2:1.02.77-6ubuntu2) ...
Setting up libicu52:amd64 (52.1-3ubuntu0.4) ...
Setting up liblzo2-2:amd64 (2.06-1.2ubuntu1.1) ...
Setting up bcache-tools (1.0.7-1~14.04.1) ...
update-initramfs: deferring update (trigger activated)
Setting up btrfs-tools (3.12-1ubuntu0.1) ...
update-initramfs: deferring update (trigger activated)
Setting up watershed (7) ...
update-initramfs: deferring update (trigger activated)
Setting up lvm2 (2.02.98-6ubuntu2) ...
update-initramfs: deferring update (trigger activated)
Setting up mdadm (3.2.5-5ubuntu4.3) ...
Generating mdadm.conf... done.
 Removing any system startup links for /etc/init.d/mdadm-raid ...
update-initramfs: deferring update (trigger activated)
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.2.0-42-generic
Found initrd image: /boot/initrd.img-4.2.0-42-generic
Found linux image: /boot/vmlinuz-4.2.0-27-generic
Found initrd image: /boot/initrd.img-4.2.0-27-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
File descriptor 3 (pipe:[35066792]) leaked on lvs invocation. Parent PID 41510: /bin/sh
  No volume groups found
done
 * Starting MD monitoring service mdadm --monitor                                                                                                                                                    [ OK ] 
Setting up postfix (2.11.0-1ubuntu1) ...
Adding group `postfix' (GID 118) ...
Done.
Adding system user `postfix' (UID 110) ...
Adding new user `postfix' (UID 110) with group `postfix' ...
Not creating home directory `/var/spool/postfix'.
Creating /etc/postfix/dynamicmaps.cf
Adding tcp map entry to /etc/postfix/dynamicmaps.cf
Adding sqlite map entry to /etc/postfix/dynamicmaps.cf
Adding group `postdrop' (GID 119) ...
Done.
/etc/aliases does not exist, creating it.

Postfix was not set up.  Start with
  cp /usr/share/postfix/main.cf.debian /etc/postfix/main.cf
.  If you need to make changes, edit
/etc/postfix/main.cf (and others) as needed.  To view Postfix configuration
values, see postconf(1).

After modifying main.cf, be sure to run '/etc/init.d/postfix reload'.

Setting up python3-crypto (2.6.1-4build1) ...
Setting up python3-oauthlib (0.6.1-1) ...
Setting up python3-yaml (3.10-4ubuntu0.1) ...
Setting up xfsprogs (3.1.9ubuntu2) ...
Setting up gdisk (0.8.8-1ubuntu0.1) ...
Setting up curtin-common (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) ...
Setting up python-curtin (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) ...
Setting up python3-curtin (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up curtin (0.1.0~bzr419-0ubuntu1~ubuntu14.04.1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
Processing triggers for initramfs-tools (0.103ubuntu4.4) ...
update-initramfs: Generating /boot/initrd.img-4.2.0-42-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Processing triggers for ufw (0.34~rc-0ubuntu2) ...

nodes are now failing to deploy in MaaS with Ubuntu Trusty and "hwe-x' kernel. Please take a look on attached screenshots (screenshots.tar.gz) for more information.

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-16:

#12

screenshots.tar.gz Edit (474.2 KiB, application/x-tar)

Revision history for this message

Ryan Harper (raharper) wrote on 2016-09-16:

#13

Can you provide the curtin config you send to the nodes?

https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b

It looks like some apt repo config is being sent and breaking/blocking
access to packages needed.

On Fri, Sep 16, 2016 at 6:46 AM, Tytus <email address hidden> wrote:

> ** Attachment added: "screenshots.tar.gz"
> https://bugs.launchpad.net/curtin/+bug/1618429/+
> attachment/4742022/+files/screenshots.tar.gz
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

Revision history for this message

Ryan Harper (raharper) wrote on 2016-09-16:

#14

The screenshot shows this "Stale file handle line" which is a bug in
kernel overlayfs:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1618572

which is what I think you're seeing now.

On Fri, Sep 16, 2016 at 8:44 AM, Ryan Harper <email address hidden>
wrote:

> Can you provide the curtin config you send to the nodes?
>
> https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b
>
> It looks like some apt repo config is being sent and breaking/blocking
> access to packages needed.
>
> On Fri, Sep 16, 2016 at 6:46 AM, Tytus <email address hidden> wrote:
>
>> ** Attachment added: "screenshots.tar.gz"
>> https://bugs.launchpad.net/curtin/+bug/1618429/+attachment/
>> 4742022/+files/screenshots.tar.gz
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1618429
>>
>> Title:
>> Curtin doesn't clean up previous MD configuration
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>>
>
>

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-19:

#15

This happens when using default curtin config.

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-19:

#16

An update. It looks like the issue I attached in "screenshots.tar.gz" file occurred after updating the boot source to point to "daily" instead of "releases", not after installing "curtin" package from your PPA.

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-19:

#17

I installed "curtin" package from "ppa:curtin-dev/daily" PPA on MaaS node, attempted to deploy with Ubuntu Trusty and "hwe-x" kernel, and the issue with mdadm still persists.

Revision history for this message

Ryan Harper (raharper) wrote on 2016-09-19:

#18

Hi, in order to further investigate this bug we'll need to get some
information about the node. Please collect and attach the storage
configuration for the node. You can get this information with:

* maas 1.9 via cli
maas <session> node get-curtin-config <system-id>
maas <session> maas set-config name=curtin_verbose value=true

* maas 2.0 via cli
maas <session> machine get-curtin-config <system-id>

* Web UI:
On the node details page in the installation output section at the bottom
of the page

The cloud-init output logs from the host that's failing are of particular
importance so we can
see if you're encountering a different issue with mdadm wiping, versus what
we've found to be
an issue with the original bug (mdadm assemble return codes when wiping
disks).

Thanks,
Ryan

On Mon, Sep 19, 2016 at 10:25 AM, Tytus <email address hidden> wrote:

> I installed "curtin" package from "ppa:curtin-dev/daily" PPA on MaaS
> node, attempted to deploy with Ubuntu Trusty and "hwe-x" kernel, and the
> issue with mdadm still persists.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-20:

#19

Download full text (9.1 KiB)

Ryan,

Please find the output below:

bootstack@optionsit-os-maas-1:~$ maas maas-root node get-curtin-config node-40b72c26-7e68-11e6-9b2c-3c4a92b2d991
Success.
Machine-readable output follows:
apt_mirrors:
  ubuntu_archive: http://archive.ubuntu.com//ubuntu
  ubuntu_security: http://archive.ubuntu.com//ubuntu
apt_proxy: http://172.18.252.11:8000/
debconf_selections:
  maas: 'cloud-init cloud-init/datasources multiselect MAAS

cloud-init cloud-init/maas-metadata-url string http://172.18.252.11/MAAS/metadata/

cloud-init cloud-init/maas-metadata-credentials string oauth_token_key=qeRbdZp4SJvdEeCXvB&oauth_token_secret=MYec62ZQ8whbDGLp6tymSjVtKD7d5S4c&oauth_consumer_key=nYFBV3arCSuvbn4cft

    cloud-init cloud-init/local-cloud-config string apt_preserve_sources_list:
    true\napt_proxy: http://172.18.252.11:8000/\nmanage_etc_hosts: false\nmanual_cache_clean:
    true\nreporting:\n maas: {consumer_key: nYFBV3arCSuvbn4cft, endpoint: ''http://172.18.252.11/MAAS/metadata/status/node-40b72c26-7e68-11e6-9b2c-3c4a92b2d991'',\n token_key:
    qeRbdZp4SJvdEeCXvB, token_secret: MYec62ZQ8whbDGLp6tymSjVtKD7d5S4c,\n type:
    webhook}\nsystem_info:\n package_mirrors:\n - arches: [i386, amd64]\n failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n search:\n primary:
    [''http://archive.ubuntu.com/ubuntu'']\n security: [''http://archive.ubuntu.com/ubuntu'']\n -
    arches: [default]\n failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n search:\n primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

    '
install:
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
kernel:
  mapping: {}
  package: linux-generic-lts-xenial
late_commands:
  maas:
  - wget
  - --no-proxy
  - http://172.18.252.11/MAAS/metadata/latest/by-id/node-40b72c26-7e68-11e6-9b2c-3c4a92b2d991/
  - --post-data
  - op=netboot_off
  - -O
  - /dev/null
network:
  config:
  - id: eth2
    mac_address: 14:02:ec:06:c8:ec
    mtu: 8900
    name: eth2
    subnets:
    - address: 172.18.252.22/23
      dns_nameservers: []
      gateway: 172.18.252.1
      type: static
    type: physical
  - id: eth0
    mac_address: 24:8a:07:0d:de:7c
    mtu: 8900
    name: eth0
    subnets:
    - type: manual
    type: physical
  - id: eth5
    mac_address: 14:02:ec:06:c8:ef
    mtu: 8900
    name: eth5
    subnets:
    - type: manual
    type: physical
  - id: eth4
    mac_address: 14:02:ec:06:c8:ee
    mtu: 8900
    name: eth4
    subnets:
    - type: manual
    type: physical
  - id: eth3
    mac_address: 14:02:ec:06:c8:ed
    mtu: 8900
    name: eth3
    subnets:
    - type: manual
    type: physical
  - id: eth1
    mac_address: 24:8a:07:0d:de:7d
    mtu: 8900
    name: eth1
    subnets:
    - type: manual
    type: physical
  - address: 172.18.252.11
    search:
    - ny4.oit-ops.com
    type: nameserver
  version: 1
network_commands:
  builtin:
  - curtin
  - net-meta
  - custom
partitioning_commands:
  builtin:
  - curtin
  - block-meta
  - custom
powe...

Ryan,

Please find the output below:

bootstack@optionsit-os-maas-1:~$ maas maas-root node get-curtin-config node-40b72c26-7e68-11e6-9b2c-3c4a92b2d991
Success.
Machine-readable output follows:
apt_mirrors:
  ubuntu_archive: http://archive.ubuntu.com//ubuntu
  ubuntu_security: http://archive.ubuntu.com//ubuntu
apt_proxy: http://172.18.252.11:8000/
debconf_selections:
  maas: 'cloud-init   cloud-init/datasources  multiselect MAAS

cloud-init   cloud-init/maas-metadata-url  string http://172.18.252.11/MAAS/metadata/

cloud-init   cloud-init/maas-metadata-credentials  string oauth_token_key=qeRbdZp4SJvdEeCXvB&oauth_token_secret=MYec62ZQ8whbDGLp6tymSjVtKD7d5S4c&oauth_consumer_key=nYFBV3arCSuvbn4cft

cloud-init   cloud-init/local-cloud-config  string apt_preserve_sources_list:
    true\napt_proxy: http://172.18.252.11:8000/\nmanage_etc_hosts: false\nmanual_cache_clean:
    true\nreporting:\n  maas: {consumer_key: nYFBV3arCSuvbn4cft, endpoint: ''http://172.18.252.11/MAAS/metadata/status/node-40b72c26-7e68-11e6-9b2c-3c4a92b2d991'',\n    token_key:
    qeRbdZp4SJvdEeCXvB, token_secret: MYec62ZQ8whbDGLp6tymSjVtKD7d5S4c,\n    type:
    webhook}\nsystem_info:\n  package_mirrors:\n  - arches: [i386, amd64]\n    failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n    search:\n      primary:
    [''http://archive.ubuntu.com/ubuntu'']\n      security: [''http://archive.ubuntu.com/ubuntu'']\n  -
    arches: [default]\n    failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n    search:\n      primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n      security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

'
install:
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
kernel:
  mapping: {}
  package: linux-generic-lts-xenial
late_commands:
  maas:
  - wget
  - --no-proxy
  - http://172.18.252.11/MAAS/metadata/latest/by-id/node-40b72c26-7e68-11e6-9b2c-3c4a92b2d991/
  - --post-data
  - op=netboot_off
  - -O
  - /dev/null
network:
  config:
  - id: eth2
    mac_address: 14:02:ec:06:c8:ec
    mtu: 8900
    name: eth2
    subnets:
    - address: 172.18.252.22/23
      dns_nameservers: []
      gateway: 172.18.252.1
      type: static
    type: physical
  - id: eth0
    mac_address: 24:8a:07:0d:de:7c
    mtu: 8900
    name: eth0
    subnets:
    - type: manual
    type: physical
  - id: eth5
    mac_address: 14:02:ec:06:c8:ef
    mtu: 8900
    name: eth5
    subnets:
    - type: manual
    type: physical
  - id: eth4
    mac_address: 14:02:ec:06:c8:ee
    mtu: 8900
    name: eth4
    subnets:
    - type: manual
    type: physical
  - id: eth3
    mac_address: 14:02:ec:06:c8:ed
    mtu: 8900
    name: eth3
    subnets:
    - type: manual
    type: physical
  - id: eth1
    mac_address: 24:8a:07:0d:de:7d
    mtu: 8900
    name: eth1
    subnets:
    - type: manual
    type: physical
  - address: 172.18.252.11
    search:
    - ny4.oit-ops.com
    type: nameserver
  version: 1
network_commands:
  builtin:
  - curtin
  - net-meta
  - custom
partitioning_commands:
  builtin:
  - curtin
  - block-meta
  - custom
power_state:
  mode: reboot
reporting:
  maas:
    consumer_key: nYFBV3arCSuvbn4cft
    endpoint: http://172.18.252.11/MAAS/metadata/status/node-40b72c26-7e68-11e6-9b2c-3c4a92b2d991
    token_key: qeRbdZp4SJvdEeCXvB
    token_secret: MYec62ZQ8whbDGLp6tymSjVtKD7d5S4c
    type: webhook
storage:
  config:
  - grub_device: true
    id: sda
    model: SAMSUNG MZ7KM1T9
    name: sda
    ptable: gpt
    serial: 5002538c00089d0b
    type: disk
    wipe: superblock
  - id: sdb
    model: SAMSUNG MZ7KM1T9
    name: sdb
    ptable: gpt
    serial: 5002538c0008a163
    type: disk
    wipe: superblock
  - id: sdc
    model: SAMSUNG MZ7KM1T9
    name: sdc
    ptable: gpt
    serial: 5002538c00089d5c
    type: disk
    wipe: superblock
  - id: sdd
    model: SAMSUNG MZ7KM1T9
    name: sdd
    ptable: gpt
    serial: 5002538c00089d08
    type: disk
    wipe: superblock
  - id: sde
    model: ST6000NM0054
    name: sde
    ptable: gpt
    serial: 5000c50084fe7a03
    type: disk
    wipe: superblock
  - id: sdf
    model: ST6000NM0054
    name: sdf
    ptable: gpt
    serial: 5000c50085120ccb
    type: disk
    wipe: superblock
  - id: sdg
    model: ST8000NM0095
    name: sdg
    ptable: gpt
    serial: 5000c50084ca417b
    type: disk
    wipe: superblock
  - id: sdh
    model: ST8000NM0095
    name: sdh
    ptable: gpt
    serial: 5000c50084c6bbc7
    type: disk
    wipe: superblock
  - device: sda
    id: sda-part1
    name: sda-part1
    number: 1
    offset: 4194304B
    size: 1920374669312B
    type: partition
    uuid: adae4a53-d43a-40cc-935d-1fb65e735641
    wipe: superblock
  - device: sdb
    id: sdb-part1
    name: sdb-part1
    number: 1
    offset: 4194304B
    size: 1920374669312B
    type: partition
    uuid: cdf5bae8-654e-4c0a-af4b-1c9b33664912
    wipe: superblock
  - device: sdc
    id: sdc-part1
    name: sdc-part1
    number: 1
    offset: 4194304B
    size: 1920374669312B
    type: partition
    uuid: dc57d3a5-e924-4a91-99c6-87ec28d3fd04
    wipe: superblock
  - device: sdd
    id: sdd-part1
    name: sdd-part1
    number: 1
    offset: 4194304B
    size: 1920374669312B
    type: partition
    uuid: b6778481-d1be-43fc-9628-516e26124667
    wipe: superblock
  - devices:
    - sda-part1
    - sdb-part1
    - sdc-part1
    - sdd-part1
    id: md4
    name: md4
    raidlevel: 0
    spare_devices: []
    type: raid
  - device: sde
    id: sde-part1
    name: sde-part1
    number: 1
    offset: 4194304B
    size: 998244352B
    type: partition
    uuid: 911738aa-b043-4c07-a5d9-9402e95d1304
    wipe: superblock
  - device: sde
    id: sde-part2
    name: sde-part2
    number: 2
    size: 998244352B
    type: partition
    uuid: b4a4f2d8-d9c3-431a-a69e-8bac986f9734
    wipe: superblock
  - device: sde
    id: sde-part3
    name: sde-part3
    number: 3
    size: 1989995921408B
    type: partition
    uuid: 42bec107-6e59-437f-b6f8-3c882f885ebe
    wipe: superblock
  - device: sde
    id: sde-part4
    name: sde-part4
    number: 4
    size: 4009175810048B
    type: partition
    uuid: 33624f3d-6a72-4774-981a-d52cb503a9e0
    wipe: superblock
  - device: sdf
    id: sdf-part1
    name: sdf-part1
    number: 1
    offset: 4194304B
    size: 998244352B
    type: partition
    uuid: 1efc6a7d-d065-41d5-a25c-4c75f1e441cc
    wipe: superblock
  - device: sdf
    id: sdf-part2
    name: sdf-part2
    number: 2
    size: 998244352B
    type: partition
    uuid: 7ac22b2b-475b-4cbb-8899-412245e1a063
    wipe: superblock
  - devices:
    - sde-part2
    - sdf-part2
    id: md0
    name: md0
    raidlevel: 1
    spare_devices: []
    type: raid
  - device: sdf
    id: sdf-part3
    name: sdf-part3
    number: 3
    size: 1989995921408B
    type: partition
    uuid: e7d7d4cb-221a-4613-b703-0a887ae69fb1
    wipe: superblock
  - devices:
    - sde-part3
    - sdf-part3
    id: md1
    name: md1
    raidlevel: 1
    spare_devices: []
    type: raid
  - device: sdf
    id: sdf-part4
    name: sdf-part4
    number: 4
    size: 4009175810048B
    type: partition
    uuid: ab69f6b0-b75d-4e48-921a-b1a5f05a447f
    wipe: superblock
  - devices:
    - sde-part4
    - sdf-part4
    id: md2
    name: md2
    raidlevel: 1
    spare_devices: []
    type: raid
  - device: sdg
    id: sdg-part1
    name: sdg-part1
    number: 1
    offset: 4194304B
    size: 8001557626880B
    type: partition
    uuid: 752f416e-0c4d-4172-84c3-47182c73ada6
    wipe: superblock
  - device: sdh
    id: sdh-part1
    name: sdh-part1
    number: 1
    offset: 4194304B
    size: 8001557626880B
    type: partition
    uuid: d0817c3e-9c7a-492c-a5ab-272fc2b76519
    wipe: superblock
  - devices:
    - sdg-part1
    - sdh-part1
    id: md3
    name: md3
    raidlevel: 1
    spare_devices: []
    type: raid
  - devices:
    - md2
    - md3
    id: vg0
    name: vg0
    type: lvm_volgroup
    uuid: d378ddd1-03ea-4978-9c98-e538f5ec567e
  - id: vg0-lv0
    name: lv0
    size: 11989996797952B
    type: lvm_partition
    volgroup: vg0
  - fstype: fat32
    id: sde-part1_format
    label: ''
    type: format
    uuid: 0504e4fc-680c-4519-96d7-927ae783a39a
    volume: sde-part1
  - fstype: fat32
    id: sdf-part1_format
    label: ''
    type: format
    uuid: ab2359ec-1542-448f-b29d-d3ca1c2ccfe5
    volume: sdf-part1
  - fstype: ext4
    id: md0_format
    label: ''
    type: format
    uuid: 64044811-e875-40e4-9f9f-8c00e829246d
    volume: md0
  - fstype: ext4
    id: md1_format
    label: ''
    type: format
    uuid: 382678d0-0687-4a46-95ec-1e7f9dfa33c4
    volume: md1
  - fstype: ext4
    id: vg0-lv0_format
    label: ''
    type: format
    uuid: 5318ad79-b310-41af-a6dd-ca144daaecbd
    volume: vg0-lv0
  - device: md1_format
    id: md1_mount
    path: /
    type: mount
  - device: md0_format
    id: md0_mount
    path: /boot
    type: mount
  - device: sde-part1_format
    id: sde-part1_mount
    path: /boot/efi
    type: mount
  - device: vg0-lv0_format
    id: vg0-lv0_mount
    path: /var/lib/nova/instances
    type: mount
  version: 1

Also, please find attached "cloud-init.log" and "cloud-init-output.log" files from the failing node.

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-20:

#20

cloud-init.log Edit (137.8 KiB, text/plain)

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-20:

#21

cloud-init-output.log Edit (15.0 KiB, text/plain)

Revision history for this message

Ryan Harper (raharper) wrote on 2016-09-20:

#22

Thanks for the configs and logs. It appears to me that it's not running
the newer curtin.
The updated curtin package runs mdadm --assemble --scan -v, which I'm not
seeing in the cloud-init-output.log

If you can obtain the install.log that curtin sends back to maas, I'm told
that's on the Node details page.
Or if you have access to the endpoint, maas configures to write it out to
/tmp/install.log.

That would be the most helpful.

On Tue, Sep 20, 2016 at 1:27 AM, Tytus <email address hidden> wrote:

> ** Attachment added: "cloud-init-output.log"
> https://bugs.launchpad.net/curtin/+bug/1618429/+
> attachment/4744100/+files/cloud-init-output.log
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

Revision history for this message

Tytus Kurek (tkurek) wrote on 2016-09-21:

#23

These nodes are no longer available for testing at the moment. I'll provide an update if anything changes here.

Ryan Harper (raharper) on 2016-10-03

description:

updated

Revision history for this message

Andy Whitcroft (apw) wrote on 2016-10-05: Please test proposed package

#24

Hello Ante, or anyone else affected,

Accepted curtin into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr425-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-needed

Andy Whitcroft (apw) on 2016-10-05

Changed in curtin (Ubuntu):
status:	New → Fix Released
Changed in curtin (Ubuntu Xenial):
status:	New → Fix Committed

Revision history for this message

Jon Grimm (jgrimm) wrote on 2016-10-12:

#25

Added José Pekkarinen, in hopes able to verify the xenial-proposed fix for this bug.

Revision history for this message

Ryan Harper (raharper) wrote on 2016-10-14:

#26

I'd like to add some comments for the SRU verification.

This issue was discovered on a customer's system which is not currently available for verification. Analyzing the error code in the original bug was when mdadm assemble was called on an array that was incomplete (some members not available) and produced a return code of '2'. The curtin code that handles mdadm assembly was not expecting a return code of '2' and exited.

After examination of the process, curtin is attempting to assemble any arrays to discover any block device dependencies in an effort to wipe them all clean (ensuring that we can install and reboot into a system configured as expected). We determined that during storage preparation curtin does not care specifically if assembly fails as it was going to *wipe* the target block devices in any case; thus adding a flag to ignore return codes from mdadm assembly during this phase was the recommended solution.

We were not able to simulate this particular failure, so instead we have a unittest which explicitly returns a value of 2 to simulate the situation and ensure that during our prepare phase we ignore those errors.

Despite not recreating this specific error, we feel confident that the fix for wiping previous md devices is fixed in this release.

Jon Grimm (jgrimm) on 2016-10-14

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-10-17:

#27

This bug was fixed in the package curtin - 0.1.0~bzr425-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr425-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

[ Scott Moser ]
* debian/new-upstream-snapshot: add writing of debian changelog entries.

  [ Ryan Harper ]
  * New upstream snapshot.
    - unittest,tox.ini: catch and fix issue with trusty-level mock of open
    - block/mdadm: add option to ignore mdadm_assemble errors (LP: #1618429)
    - curtin/doc: overhaul curtin documentation for readthedocs.org
      (LP: #1351085)
    - curtin.util: re-add support for RunInChroot (LP: #1617375)
    - curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
    - curtin.block: refactor clear_holders logic into block.clear_holders and
      cli cmd
    - curtin.apply_net should exit non-zero upon exception. (LP: #1615780)
    - apt: fix bug in disable_suites if sources.list line is blank.
    - vmtests: disable Wily in vmtests
    - Fix the unittests for test_apt_source.
    - get CURTIN_VMTEST_PARALLEL shown correctly in jenkins-runner output
    - fix vmtest check_file_strippedline to strip lines before comparing
    - fix whitespace damage in tests/vmtests/__init__.py
    - fix dpkg-reconfigure when debconf_selections was provided.
      (LP: #1609614)
    - fix apt tests on non-intel arch
    - Add apt features to curtin. (LP: #1574113)
    - vmtest: easier use of parallel and controlling timeouts
    - mkfs.vfat: add force flag for formating whole disks (LP: #1597923)
    - block.mkfs: fix sectorsize flag (LP: #1597522)
    - block_meta: cleanup use of sys_block_path and handle cciss knames
      (LP: #1562249)
    - block.get_blockdev_sector_size: handle _lsblock multi result return
      (LP: #1598310)
    - util: add target (chroot) support to subp, add target_path helper.
    - block_meta: fallback to parted if blkid does not produce output
      (LP: #1524031)
    - commands.block_wipe: correct default wipe mode to 'superblock'
    - tox.ini: run coverage normally rather than separately
    - move uefi boot knowledge from launch and vmtest to xkvm

-- Ryan Harper <email address hidden> Mon, 03 Oct 2016 13:43:54 -0500

This bug was fixed in the package curtin - 0.1.0~bzr425-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr425-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

[ Scott Moser ]
  * debian/new-upstream-snapshot: add writing of debian changelog entries.

[ Ryan Harper ]
  * New upstream snapshot.
    - unittest,tox.ini: catch and fix issue with trusty-level mock of open
    - block/mdadm: add option to ignore mdadm_assemble errors  (LP: #1618429)
    - curtin/doc: overhaul curtin documentation for readthedocs.org
      (LP: #1351085)
    - curtin.util: re-add support for RunInChroot  (LP: #1617375)
    - curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
    - curtin.block: refactor clear_holders logic into block.clear_holders and
      cli cmd
    - curtin.apply_net should exit non-zero upon exception.  (LP: #1615780)
    - apt: fix bug in disable_suites if sources.list line is blank.
    - vmtests: disable Wily in vmtests
    - Fix the unittests for test_apt_source.
    - get CURTIN_VMTEST_PARALLEL shown correctly in jenkins-runner output
    - fix vmtest check_file_strippedline to strip lines before comparing
    - fix whitespace damage in tests/vmtests/__init__.py
    - fix dpkg-reconfigure when debconf_selections was provided.
      (LP: #1609614)
    - fix apt tests on non-intel arch
    - Add apt features to curtin.  (LP: #1574113)
    - vmtest: easier use of parallel and controlling timeouts
    - mkfs.vfat: add force flag for formating whole disks  (LP: #1597923)
    - block.mkfs: fix sectorsize flag  (LP: #1597522)
    - block_meta: cleanup use of sys_block_path and handle cciss knames
      (LP: #1562249)
    - block.get_blockdev_sector_size: handle _lsblock multi result return
      (LP: #1598310)
    - util: add target (chroot) support to subp, add target_path helper.
    - block_meta: fallback to parted if blkid does not produce output
      (LP: #1524031)
    - commands.block_wipe:  correct default wipe mode to 'superblock'
    - tox.ini: run coverage normally rather than separately
    - move uefi boot knowledge from launch and vmtest to xkvm

-- Ryan Harper <ryan.harper@canonical.com>  Mon, 03 Oct 2016 13:43:54 -0500

Changed in curtin (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Revision history for this message

Martin Pitt (pitti) wrote on 2016-10-17: Update Released

#28

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Ante Karamatić (ivoks) wrote on 2017-01-18:

#29

curtin-config.txt Edit (5.7 KiB, text/plain)

I'm not sure this is fixed. We are still seeing problems:

MAAS' curtin version is 0.1.0~bzr425-0ubuntu1~16.04.1. Installation log and curtin-config (obfuscated) is provided as attachments.

Revision history for this message

Ante Karamatić (ivoks) wrote on 2017-01-18:

#30

installation.log Edit (2.9 KiB, text/plain)

Revision history for this message

Ryan Harper (raharper) wrote on 2017-01-18: Re: [Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

#31

Is it possible to include the entire installation.log?

Do repeat deploys to the same node and same configuration fail consistently?

On Wed, Jan 18, 2017 at 6:50 AM, Ante Karamatić <
<email address hidden>> wrote:

> ** Attachment added: "installation.log"
> https://bugs.launchpad.net/curtin/+bug/1618429/+
> attachment/4805810/+files/installation.log
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

Revision history for this message

Ante Karamatić (ivoks) wrote on 2017-01-18:

#32

Yes, failures are consistent and happen even after 'erase disks' is enabled
in MAAS. They also happen on all nodes. I'm afraid I can't get any more
logs than this. This is all that MAAS can capture in this environment.

On Wed, Jan 18, 2017 at 3:00 PM Ryan Harper <email address hidden>
wrote:

> Is it possible to include the entire installation.log?
>
> Do repeat deploys to the same node and same configuration fail
> consistently?
>
> On Wed, Jan 18, 2017 at 6:50 AM, Ante Karamatić <
> <email address hidden>> wrote:
>
> > ** Attachment added: "installation.log"
> > https://bugs.launchpad.net/curtin/+bug/1618429/+
> > attachment/4805810/+files/installation.log
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> > Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>
--
Ante Karamatić
<email address hidden>
Canonical

Revision history for this message

Ryan Harper (raharper) wrote on 2017-01-18:

#33

Download full text (3.2 KiB)

On Wed, Jan 18, 2017 at 8:43 AM, Ante Karamatić <
<email address hidden>> wrote:

> Yes, failures are consistent and happen even after 'erase disks' is enabled
> in MAAS. They also happen on all nodes. I'm afraid I can't get any more
> logs than this. This is all that MAAS can capture in this environment.
>

Hrm, it would be *really* helpful to have the complete install log. In
particular, curtin dumps
out the storage relationship of the disks before attempting to apply the
storage config

For example, when I attempt to recreate this by redeploying the same config
to the same disks
I can see this output:

Current device storage tree:
vdc
|-- vdc1
|-- vdc2
| `-- md0
`-- vdc3
`-- md1
vdd
|-- vdd1
|-- vdd2
| `-- md0
`-- vdd3
`-- md1

Which shows the relationship that was discovered and cleared. We then clear
the tree from the bottom up; wiping md superblocks and metadata,
wiping partitions, and then devices. Resulting in a successful clearing
out of
previous on-disk storage configuration. And install succeeds. Something
is different between these systems and lack of debug info is going to
prevent
us from coming to a resolution.

The full log details the steps and the lists of current block device
holders.

If we have access to a node, then I'd like to get a node in failed state to
inspect.

https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b

That has details on how to stop a deployment node:

ssh ubuntu@$HOST_IP sudo touch /run/block-curtin-poweroff

If you can, then after a failed curtin install, getting at
/var/log/curtin/install.log
will contain everything we need to investigate further.

> On Wed, Jan 18, 2017 at 3:00 PM Ryan Harper <email address hidden>
> wrote:
>
> > Is it possible to include the entire installation.log?
> >
> > Do repeat deploys to the same node and same configuration fail
> > consistently?
> >
> > On Wed, Jan 18, 2017 at 6:50 AM, Ante Karamatić <
> > <email address hidden>> wrote:
> >
> > > ** Attachment added: "installation.log"
> > > https://bugs.launchpad.net/curtin/+bug/1618429/+
> > > attachment/4805810/+files/installation.log
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1618429
> > >
> > > Title:
> > > Curtin doesn't clean up previous MD configuration
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > >
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> > Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
> --
> Ante Karamatić
> <email address hidden>
> Canonical
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
> Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/...

curtin

Curtin doesn't clean up previous MD configuration

Bug Description

Related branches

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

	Status	Importance	Assigned to
curtin	Fix Released	High	Unassigned
curtin (Ubuntu)	Fix Released	Undecided	Unassigned
Xenial	Fix Released	Undecided	Unassigned