Re-targeted to Juju (2.6.5) based on https://bugs.launchpad.net/maas/+bug/1835823/comments/2 (see this comment for details)
Old title: "maas reported 409 CONFLICT from "allocate" while the node matching constraints was available"
https://solutions.qa.canonical.com/#/qa/testRun/808239c1-7b06-4281-9267-0f09421604a1
https://oil-jenkins.canonical.com/artifacts/808239c1-7b06-4281-9267-0f09421604a1/index.html (deployment artifacts)
Juju tried 10 times to allocate a machine with tag "prometheus" but MAAS responded with 509s to all requests so Juju eventually marked the machine as failed. The machine is a pod VM (the host is in zone-3). Based on the DB dump it looks like it exists, is ready, has the right tag and interface with access to the right space.
8cf17ecd-809f-4c14-8220-09b6b66bbc74: machine-0 2019-07-05 19:55:24 WARNING juju.worker.provisioner provisioner_task.go:1157 failed to start machine 3 (failed to acquire node: No available machine matches constraints: [('agent_name', ['85f7eddb-f1bb-45fd-8eba-e9e50b532eda']), ('interfaces', ['internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3']), ('tags', ['prometheus']), ('zone', ['zone3'])] (resolved to "interfaces=internal:space=2;nrpe-external-master:space=3;target:space=3;compute-peer:space=3;prometheus-rules:space=3;website:space=3;amqp:space=3;secrets-storage:space=3;neutron-plugin:space=3;nova-ceilometer:space=3;cloud-compute:space=3;ephemeral-backend:space=3;snmp-exporter:space=3;grafana-source:space=3;image-service:space=3;lxd:space=3;scrape:space=3;blackbox-exporter:space=3;cloud-credentials:space=3;ceph-access:space=3;0:space=3;alertmanager-service:space=3;ceph:space=3 tags=prometheus zone=zone3")), retrying in 10s (10 more attempts)
2019-07-05 19:55:23 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 200 OK (referrer: -; agent: Go-http-client/1.1)
2019-07-05 19:55:24 regiond: [info] 10.246.64.6 POST /MAAS/api/2.0/machines/?op=allocate HTTP/1.1 --> 409 CONFLICT (referrer: -; agent: Go-http-client/1.1)
select x.hostname,x.status,x.zone_id,x.name from (select * from maasserver_node inner join maasserver_zone on maasserver_node.zone_id = maasserver_zone.id) as x where x.hostname LIKE '%prometheus%';
hostname | status | zone_id | name
--------------+--------+---------+-------
prometheus-3 | 4 | 3 | zone3
(1 row)
src/maasserver/enum.py
class NODE_STATUS:
# ...
READY = 4
#: The node is ready for named deployment.
maasdb=# select id from maasserver_node where hostname = 'prometheus-3';
id
----
13
(1 row)
maasdb=# select * from maasserver_space;
id | created | updated | name | description
----+-------------------------------+-------------------------------+--------------------+-----------------------------------------------------
1 | 2019-07-05 19:02:02.797481+00 | 2019-07-05 19:02:02.797481+00 | ps45_routers | foundation-engine created space: ps45_routers
2 | 2019-07-05 19:02:03.603296+00 | 2019-07-05 19:02:03.603296+00 | internal-space | foundation-engine created space: internal-space
3 | 2019-07-05 19:02:05.074262+00 | 2019-07-05 19:02:05.074262+00 | oam-space | foundation-engine created space: oam-space
4 | 2019-07-05 19:02:06.108567+00 | 2019-07-05 19:02:06.108567+00 | external-space | foundation-engine created space: external-space
5 | 2019-07-05 19:02:06.922964+00 | 2019-07-05 19:02:06.922964+00 | ceph-replica-space | foundation-engine created space: ceph-replica-space
6 | 2019-07-05 19:02:07.80897+00 | 2019-07-05 19:02:07.80897+00 | ceph-access-space | foundation-engine created space: ceph-access-space
(6 rows)
maasdb=# select vlan_id from maasserver_interface where node_id = 13;
vlan_id
---------
5002
(1 row)
maasdb=# select id,space_id from maasserver_vlan where id = 5002;
id | space_id
------+----------
5002 | 3
(1 row)
maasdb=# select node_id,tag_id from maasserver_node inner join maasserver_node_tags on maasserver_node.id = maasserver_node_tags.node_id where maasserver_node.id = 13;
node_id | tag_id
---------+--------
13 | 2
13 | 1
13 | 9
(3 rows)
maasdb=# select * from maasserver_tag where name like '%prometheus%';
id | created | updated | name | definition | comment | kernel_opts
----+-------------------------------+-------------------------------+------------+------------+---------+-------------
9 | 2019-07-05 19:03:58.501961+00 | 2019-07-05 19:03:58.501961+00 | prometheus | | |
(1 row)
Dmitrii,
Can you verify that the machine also matches the interface constraints that you are trying to allocate the machine with? From your constraints, interfaces need to match:
'internal: