Storage remains in pending state

Bug #1677001 reported by Jorge Niedbalski
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Andrew Wilkins
2.0
Won't Fix
Undecided
Unassigned
2.1
Won't Fix
Undecided
Unassigned

Bug Description

[Environment]

Reproduced on 2.0.3 up to 2.2-beta1-xenial-amd64
Xenial 16.04
MAAS 2.2.0 beta 3

[Description]

Isn't possible to attach block storage devices. The storage devices remain stuck
in pending state with no clear feedback to the user in the logs about what's currently
being performed.

Current Storage pools:

ubuntu@juju-client:~$ juju list-storage-pools
Name Provider Attrs
loop loop
maas maas
rootfs rootfs
sata maas tags=rotary
tmpfs tmpfs

Machine storage configuration in maas: https://ibin.co/w800/3H9D3vN5tK6c.jpg

[Reproducer]

1) Configure the machine storage in MAAS as indicated on the previous image.
2) Create the storage pool:

ubuntu@juju-client:~$ juju create-storage-pool sata maas tags=rotary

3) Set the logging config to trace.

ubuntu@juju-client:~$ juju model-config logging-config="<root>=TRACE;unit=TRACE"

4) Deploy any charm that supports storage:

ubuntu@juju-client:~$ juju deploy cs:xenial/ceph-osd --storage osd-devices=sata,1 ceph-osd

Expected: Available and attached block devices.

Current results:

ubuntu@juju-client:~$ juju list-storage
[Storage]
Unit Id Type Pool Provider id Size Status Message
ceph-osd/5 osd-devices/7 block sata volume-7 20GiB pending

ubuntu@juju-client:~$ juju status
Model Controller Cloud/Region Version
default storage-controller internal-maas 2.2-beta1

App Version Status Scale Charm Store Rev OS Notes
ceph-osd waiting 0/1 ceph-osd jujucharms 241 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-osd/5* waiting allocating 7 10.10.10.6 agent initializing

Machine State DNS Inst id Series AZ Message
7 started 10.10.10.6 aqryre xenial default Deployed

Logs:

controller logs:

http://paste.ubuntu.com/24269342/

ceph-osd/5 unit logs:

http://paste.ubuntu.com/24269348/

maas logs for machine:

http://paste.ubuntu.com/24269357/

Tags: sts
tags: added: sts
Revision history for this message
Anastasia (anastasia-macmood) wrote :

Marking as Wont' Fix for 2.0 and 2.1 since we are not planning to release another versions in these series.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

We cannot tell anything from these logs. The MAAS logs don't include the /machines/?op=allocate request, and the controller logs don't contain the storage provisioner actions. Can you please provide the full logs?

Changed in juju:
status: New → Incomplete
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

@axw I just send you some extra logs from another installation that is experiencing the same issue reported here.

- Aren't you able to reproduce the issue with the steps exposed on this bug? If not,
please let me know to clarify any step, as I am able to reproduce this all the time.

Changed in juju:
status: Incomplete → New
Changed in juju:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Andrew Wilkins (axwalk) wrote :

Jorge, I do not have a readily available MAAS to test this on. As you can easily reproduce the issue, I'm asking you for the information to diagnose.

Re extra logs, I am unable to access the system that you sent me the link to. If the contents are sensitive, please put them on private-fileshare.

Changed in juju:
status: Triaged → Incomplete
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

@axwalk, I sent you a couple of extra logs via private-fileshare for you to review.

Changed in juju:
status: Incomplete → New
Revision history for this message
Andrew Wilkins (axwalk) wrote :

Jorge, this sounds suspiciously like lp:1674148. Unfortunately the logs don't have the detail required, as the logging config was not set on the controller model. To confirm that this is the same problem, please do the following:

    juju model-config -m controller logging-config='<root>=DEBUG;unit=TRACE;maas=TRACE'

(setting <root>=TRACE would be so overwhelming as to be unhelpful)

Changed in juju:
status: New → Incomplete
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :
Changed in juju:
status: Incomplete → New
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

@axw

Hello, I just reproduced this bug with a new installation, same behavior as expressed on this bug description.

Attached you can find the Juju controller logs in TRACE level, as well
as the MAAS logs.

Please let me know if all the important information is contained on the
attached files.

Thanks.

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Additional information that might be relevant.

ubuntu@maas-master:~$ sudo lsblk --exclude 1,2,7 -d -P -o NAME,RO,RM,MODEL,ROTA,MAJ:MIN,SERIAL -x MAJ:MIN
NAME="vda" RO="0" RM="0" MODEL="" ROTA="1" MAJ:MIN="253:0" SERIAL=""
NAME="vdb" RO="0" RM="0" MODEL="" ROTA="1" MAJ:MIN="253:16" SERIAL=""

maas machine get-curtin-config : https://pastebin.canonical.com/185301/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

From the log output, it looks like it's not the same issue.

Revision history for this message
Andrew Wilkins (axwalk) wrote :
Download full text (14.5 KiB)

I think the issue is that the block device path reported by MAAS is not seen by Juju on the machine. From the logsink.log, I see:

647f1619-dab3-4347-8a9c-0418a009e664 machine-0: 2017-03-17 18:54:49 INFO juju.provisioner provisioner_task.go:758 started machine 32 as instance 4y3mp7 with hardware "arch=amd64 cores=32 mem=257859M tags=ceph-osd,gluster availability-zone=default", network config [{DeviceIndex:0 MACAddress:0c:c4:7a:8e:ee:1b CIDR: MTU:0 ProviderId:3936 ProviderSubnetId: ProviderSpaceId: ProviderAddressId: ProviderVLANId: VLANTag:0 InterfaceName:enp4s0f3 ParentInterfaceName:bond1 InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType: Address: DNSServers:[] DNSSearchDomains:[] GatewayAddress:} {DeviceIndex:1 MACAddress:0c:c4:7a:8e:ee:19 CIDR: MTU:0 ProviderId:3933 ProviderSubnetId: ProviderSpaceId: ProviderAddressId: ProviderVLANId: VLANTag:0 InterfaceName:enp4s0f1 ParentInterfaceName:bond0 InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType: Address: DNSServers:[] DNSSearchDomains:[] GatewayAddress:} {DeviceIndex:2 MACAddress:0c:c4:7a:8e:ee:1a CIDR: MTU:0 ProviderId:3935 ProviderSubnetId: ProviderSpaceId: ProviderAddressId: ProviderVLANId: VLANTag:0 InterfaceName:enp4s0f2 ParentInterfaceName:bond1 InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType: Address: DNSServers:[] DNSSearchDomains:[] GatewayAddress:} {DeviceIndex:3 MACAddress:0c:c4:7a:35:2d:3a CIDR:10.189.69.0/25 MTU:1500 ProviderId:3931 ProviderSubnetId:2 ProviderSpaceId:0 ProviderAddressId:23039 ProviderVLANId:5004 VLANTag:0 InterfaceName:enp5s0f0 ParentInterfaceName: InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType:static Address: DNSServers:[10.189.69.5] DNSSearchDomains:[] GatewayAddress:10.189.69.1} {DeviceIndex:4 MACAddress:0c:c4:7a:8e:ed:1c CIDR: MTU:0 ProviderId:3937 ProviderSubnetId: ProviderSpaceId: ProviderAddressId: ProviderVLANId: VLANTag:0 InterfaceName:enp133s0f0 ParentInterfaceName:bond0 InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType: Address: DNSServers:[] DNSSearchDomains:[] GatewayAddress:} {DeviceIndex:5 MACAddress:0c:c4:7a:8e:ed:1d CIDR: MTU:0 ProviderId:3938 ProviderSubnetId: ProviderSpaceId: ProviderAddressId: ProviderVLANId: VLANTag:0 InterfaceName:enp133s0f1 ParentInterfaceName:bond0 InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType: Address: DNSServers:[] DNSSearchDomains:[] GatewayAddress:} {DeviceIndex:6 MACAddress:0c:c4:7a:8e:ed:1e CIDR: MTU:0 ProviderId:3939 ProviderSubnetId: ProviderSpaceId: ProviderAddressId: ProviderVLANId: VLANTag:0 InterfaceName:enp133s0f2 ParentInterfaceName:bond1 InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType: Address: DNSServers:[] DNSSearchDomains:[] GatewayAddress:} {DeviceIndex:7 MACAddress:0c:c4:7a:8e:ed:1f CIDR: MTU:0 ProviderId:3940 ProviderSubnetId: ProviderSpaceId: ProviderAddressId: ProviderVLANId: VLANTag:0 InterfaceName:enp133s0f3 ParentInterfaceName:bond1 InterfaceType:ethernet Disabled:false NoAutoStart:false ConfigType: Address: DNSServers:[] DNSSearchDomains:[] GatewayAddress:} {DeviceIndex:8 MACAddress:0c:c4:7a:35:2d:3b CIDR: MTU:0 ProviderId:3934 ProviderSubnetId: ProviderSpaceId: Provid...

Changed in juju:
status: New → Triaged
milestone: none → 2.2-beta3
Revision history for this message
Andrew Wilkins (axwalk) wrote :

I can reproduce the issue on finfolk-vmaas.

Changed in juju:
status: Triaged → In Progress
assignee: nobody → Andrew Wilkins (axwalk)
Revision history for this message
Andrew Wilkins (axwalk) wrote :
Changed in juju:
status: In Progress → Fix Committed
Revision history for this message
Ante Karamatić (ivoks) wrote :

I can stil reproduce this with MAAS 2.2rc3 and juju2.2b3.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Ante, I'll need to get access to your CPE lab environment again then. Please let me know when I can test there.

Changed in juju:
status: Fix Committed → Triaged
milestone: 2.2-beta3 → none
Revision history for this message
Andrew Wilkins (axwalk) wrote :

MAAS is reporting the WWN path, and we're expecting the vendor+model+serial. I'll have to add WWN to the list of block device properties we record.

Changed in juju:
status: Triaged → In Progress
milestone: none → 2.2-beta4
Revision history for this message
Andrew Wilkins (axwalk) wrote :
Andrew Wilkins (axwalk)
Changed in juju:
status: In Progress → Fix Committed
Revision history for this message
Ante Karamatić (ivoks) wrote :

I can confirm this is fixed in 2.2b4.

Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.