can't create floating IP pools from CLI

When working from the CLI, an admin cannot create a pool of floating IPs and allocate them to a tenant without first becoming a part of that tenant. A pool of public IPs cannot be made "shared" from neutron, and the neutron command to create a new floating IP shows a failure (but *does* actually allocate the IP).

showing the work:

First, we create a tenant and a user and a public key - you probably already have all this:

[<email address hidden>] ~ $ keystone tenant-create --name dsmith --description "test tenant for dsmith" --enabled true
+-------------+----------------------------------+
| Property | Value |
+-------------+----------------------------------+
| description | test tenant for dsmith |
| enabled | True |
| id | fa816605d3b34ed494e333340ce583f8 |
| name | dsmith |
+-------------+----------------------------------+

[<email address hidden>] ~ $ keystone user-create --name dsmith --tenant dsmith --pass f00b4r --email <email address hidden> --enabled true
+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| email | <email address hidden> |
| enabled | True |
| id | aa45fd675d0f449f8910386a8341550d |
| name | dsmith |
| tenantId | fa816605d3b34ed494e333340ce583f8 |
+----------+----------------------------------+
[<email address hidden>] ~ $

I update my credentials.sh file with the new values and source the new file. I'm now the user 'dsmith', and can create myself a key pair for use in new VMs:
[<email address hidden>] ~ $ nova keypair-add dsmith_testing
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAygZiLYnDx9wuCdCVeQ0MKwtjFYz6TzbhLKbN5L5mJfc8w5Pg
 <deletia>
Xe3R2K9Bo6C//aoDlytlZCVFbaYI6R0lNMxy2Rl6EgM46L3zPwJlIA==
-----END RSA PRIVATE KEY-----

[<email address hidden>] ~ $

Now I load the new private key into my ssh keychain using 'ssh-add'. Let's just check on that new user, shall we?

[<email address hidden>] ~ $ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[<email address hidden>] ~ $

Good, that works - can we boot a machine?

[<email address hidden>] ~ $ nova boot --flavor m1.small --image precise --key-name dsmith_testing dsmith-testing
+-----------------------------+------------------------------------------------+
| Property | Value |
+-----------------------------+------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | ocs |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | JHgmrFUt2hR7 |
| config_drive | |
| created | 2014-07-01T00:29:00Z |
| flavor | m1.small (11) |
| hostId | |
| id | ba15ec30-d197-4915-b62a-198f959137b3 |
| image | precise (103e9f82-e753-40b7-a9d6-afa33fc11fdf) |
| key_name | dsmith_testing |
| metadata | {} |
| name | dsmith-testing |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | fa816605d3b34ed494e333340ce583f8 |
| updated | 2014-07-01T00:29:01Z |
| user_id | aa45fd675d0f449f8910386a8341550d |
+-----------------------------+------------------------------------------------+
[<email address hidden>] ~ $ nova list
+--------------------------------------+----------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+----------+
| ba15ec30-d197-4915-b62a-198f959137b3 | dsmith-testing | ERROR | - | NOSTATE | |
+--------------------------------------+----------------+--------+------------+-------------+----------+
[<email address hidden>] ~ $

...that looks like a no. Of course not! We're on an OpenContrail cloud, and haven't specified any networks yet.
Let's start with declaring a couple of networks with neutron:

[<email address hidden>] ~ $ neutron net-create DREW_0
Created a new network:
+-------------------------+--------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------+
| admin_state_up | True |
| contrail:fq_name | default-domain |
| | dsmith |
| | DREW_0 |
| contrail:instance_count | 0 |
| id | b2edf3c1-3f5d-4d83-9b5f-03dd722bc30d |
| name | DREW_0 |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | fa816605d3b34ed494e333340ce583f8 |
+-------------------------+--------------------------------------+
[<email address hidden>] ~ $ neutron net-create DREW_1
Created a new network:
+-------------------------+--------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------+
| admin_state_up | True |
| contrail:fq_name | default-domain |
| | dsmith |
| | DREW_1 |
| contrail:instance_count | 0 |
| id | 79cb86d0-000a-4520-8e2a-4ecb36b8a8ad |
| name | DREW_1 |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | fa816605d3b34ed494e333340ce583f8 |
+-------------------------+--------------------------------------+
[<email address hidden>] ~ $

Now we're going to need to add subnets to those networks, assuming we want to be able to have machines able to come up and talk to them. I'm not sure exactly why, but I found that the 'neutron subnet-create' command didn't much care for the fact that I named the networks DREW_0 and DREW_1, so I had to create the subnets by referencing the networks by id instead of name:
[<email address hidden>] ~ $ neutron subnet-create b2edf3c1-3f5d-4d83-9b5f-03dd722bc30d 192.168.42.0/24

Created a new subnet:
+-------------------------+----------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------+
| allocation_pools | {"start": "192.168.42.1", "end": "192.168.42.253"} |
| cidr | 192.168.42.0/24 |
| contrail:instance_count | 0 |
| contrail:ipam_fq_name | default-domain |
| | default-project |
| | default-network-ipam |
| dns_nameservers | 169.254.169.254 |
| enable_dhcp | False |
| gateway_ip | 192.168.42.254 |cd
| host_routes | {"destination": "TODO-destination", "nexthop": "TODO-nexthop"} |
| id | 6ae4b999-36e3-44f2-9f9b-69e5a3d4f705 |
| ip_version | 4 |
| name | |
| network_id | b2edf3c1-3f5d-4d83-9b5f-03dd722bc30d |
| tenant_id | fa816605d3b34ed494e333340ce583f8 |
+-------------------------+----------------------------------------------------------------+
[<email address hidden>] ~ $ neutron subnet-create 79cb86d0-000a-4520-8e2a-4ecb36b8a8ad 192.168.23.0/24
Created a new subnet:
+-------------------------+----------------------------------------------------------------+
| Field | Value |
+-------------------------+----------------------------------------------------------------+
| allocation_pools | {"start": "192.168.23.1", "end": "192.168.23.253"} |
| cidr | 192.168.23.0/24 |
| contrail:instance_count | 0 |
| contrail:ipam_fq_name | default-domain |
| | default-project |
| | default-network-ipam |
| dns_nameservers | 169.254.169.254 |
| enable_dhcp | False |
| gateway_ip | 192.168.23.254 |
| host_routes | {"destination": "TODO-destination", "nexthop": "TODO-nexthop"} |
| id | 04a69f5d-9e5c-4125-a0c8-e929d1c69451 |
| ip_version | 4 |
| name | |
| network_id | 79cb86d0-000a-4520-8e2a-4ecb36b8a8ad |
| tenant_id | fa816605d3b34ed494e333340ce583f8 |
+-------------------------+----------------------------------------------------------------+
[<email address hidden>] ~ $

... switch out of blogger mode. Sorry, this was all meant to be a blog post, and it's ending with a bug.

Ok, so here is where things start to fall apart a little. We need to basically boot machines into these networks with 'nova boot', and then we need to assign a floating IP to at least one of them so we can get onto the networks that we've created... but the actual assignment of the floating IP is where things begin failing.

[<email address hidden>] ~/WORK/jlab $ nova boot --flavor m1.small --image precise --key-name dsmith_testing --nic net-id=59bbc517-86c0-4820-8e5d-dace384370e6 --nic net-id=ea29ef39-0ce4-4c44-854e-71bdc1710ae8 "drew testing #1"
+-----------------------------+------------------------------------------------+
| Property | Value |
+-----------------------------+------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | ocs |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | jZQHRvp2gD47 |
| config_drive | |
| created | 2014-07-03T01:13:34Z |
| flavor | m1.small (11) |
| hostId | |
| id | de41d257-7efb-4b54-bf8b-c0bad9bcfa61 |
| image | precise (a3434bfd-98bd-477d-9cb5-cfd2b56e2776) |
| key_name | dsmith_testing |
| metadata | {} |
| name | drew testing #1 |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 01ae248eed154152b6bf991d1adac969 |
| updated | 2014-07-03T01:13:35Z |
| user_id | bbd0bc3b5e4e46b9ab1036704887f568 |
+-----------------------------+------------------------------------------------+

[<email address hidden>] ~/WORK/jlab $

So there's machine #1, and we'll do the same again for another one, and now 'nova list' shows us this:

[<email address hidden>] ~/WORK/jlab $ nova list
+--------------------------------------+-----------------+--------+------------+-------------+----------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------+--------+------------+-------------+----------------------------------------------+
| de41d257-7efb-4b54-bf8b-c0bad9bcfa61 | drew testing #1 | ACTIVE | - | Running | DREW-1=192.168.23.253; DREW-0=192.168.42.253 |
| 5dc71837-bb33-485b-85af-36b821c73f15 | drew testing #2 | ACTIVE | - | Running | DREW-1=192.168.23.252; DREW-0=192.168.42.252 |
+--------------------------------------+-----------------+--------+------------+-------------+----------------------------------------------+

[<email address hidden>] ~/WORK/jlab $

...but we can't actually connect to those until we assign a floating IP. So let's try that:

[<email address hidden>] ~/WORK/jlab $ nova floating-ip-list
+----+-----------+----------+------+
| Ip | Server Id | Fixed Ip | Pool |
+----+-----------+----------+------+
+----+-----------+----------+------+

[<email address hidden>] ~/WORK/jlab $

nope, nothing there... make one maybe?

[<email address hidden>] ~/WORK/jlab $ nova floating-ip-create Public
ERROR: FloatingIpPoolNotFound: Floating ip pool not found. (HTTP 404) (Request-ID: req-4a5aeff4-474b-42ab-96b6-45747ba4356f)

[<email address hidden>] ~/WORK/jlab $

Ok, so basically we need access to the floating ip pool "Public". Let's try it as admin instead. First I deleted all the floating IPs in the pool, just so we can start from scratch:

[<email address hidden>] ~/WORK/jlab $ for i in `neutron floatingip-list | grep 10 | cut -d " " -f2`; do neutron floatingip-delete $i; done
Deleted floatingip: 609b3cfa-c7c0-438d-879c-38604ef2c9f9
Deleted floatingip: 28571f0f-6045-4807-944f-9f03da2990bc
Deleted floatingip: a2d31f7b-86e8-4a66-9753-b5b9ecdfab6c
Deleted floatingip: 1eebccb8-6bc2-4ee3-ab9b-c358e35eee03
Deleted floatingip: 27a10f46-b2f7-4293-bece-dd312ed34482
...

Now we have no floating IPs in the pool, though the network is still available to our admin tenant. Check for sure:

[<email address hidden>] ~/WORK/jlab $ neutron floatingip-list

...nothing.

Ok, let's create one then!

[<email address hidden>] ~/WORK/jlab $ nova floating-ip-create public
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-e1a183db-c4fb-4012-b7fb-d807a157798f)

The command fails. OR DOES IT??

[<email address hidden>] ~/WORK/jlab $ neutron floatingip-list
+--------------------------------------+------------------+---------------------+---------+
| id | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+---------+
| fe516956-924f-41e3-a7fa-7c2ae65a2098 | | 10.129.0.253 | |
+--------------------------------------+------------------+---------------------+---------+

Looks like it just says it fails.

Anyway. Clearly we can create floating IPs from the Public network. If the 'dsmith' user/tenant could see that network, probably he could create floating IPs from it also... so let's check the docs.

According to this page: http://www.juniper.net/techpubs/en_US/contrail1.0/information-products/topic-collections/release-notes/index.html?jd0e53.html

"Shared" networks should be able to be used by other tenants or projects. If we view the "Public" network as admin, it looks like this:

[<email address hidden>] ~/WORK/jlab $ neutron net-show b09fd0c8-9f81-4a9c-8501-e129843a05d4
+-------------------------+------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------------------+
| admin_state_up | True |
| contrail:fq_name | default-domain |
| | admin |
| | Public |
| contrail:instance_count | 0 |
| contrail:subnet_ipam | {"subnet_cidr": "10.129.0.0/24", "ipam_fq_name": ["default-domain", "admin", "My-IPAM"]} |
| id | b09fd0c8-9f81-4a9c-8501-e129843a05d4 |
| name | Public |
| shared | False |
| status | ACTIVE |
| subnets | 5da81cd2-32b4-4e95-b957-f2db0e6e72ed |
| tenant_id | 29098dc789f14024be8f83c4e4cc6bc3 |
+-------------------------+------------------------------------------------------------------------------------------+

Shared = False. So, let's try updating that to be shared, according to the docs:

[<email address hidden>] ~/WORK/jlab $ neutron net-update b09fd0c8-9f81-4a9c-8501-e129843a05d4 --shared
Updated network: b09fd0c8-9f81-4a9c-8501-e129843a05d4

aaaand check it again?

[<email address hidden>] ~/WORK/jlab $ neutron net-show b09fd0c8-9f81-4a9c-8501-e129843a05d4
+-------------------------+------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------------------+
| admin_state_up | True |
| contrail:fq_name | default-domain |
| | admin |
| | Public |
| contrail:instance_count | 0 |
| contrail:subnet_ipam | {"subnet_cidr": "10.129.0.0/24", "ipam_fq_name": ["default-domain", "admin", "My-IPAM"]} |
| id | b09fd0c8-9f81-4a9c-8501-e129843a05d4 |
| name | Public |
| shared | False |
| status | ACTIVE |
| subnets | 5da81cd2-32b4-4e95-b957-f2db0e6e72ed |
| tenant_id | 29098dc789f14024be8f83c4e4cc6bc3 |
+-------------------------+------------------------------------------------------------------------------------------+

No change. The tool does not update the shared status, for some reason. Ok, back to the docs:

"Using Router:External Attribute
When a network has the router:external attribute set, users (including non-admin) in other tenants or projects can use that network for allocating floating-ips, using:
neutron floatingip-create <router-external-net-id>
then associating the IP address pool with their instances."

So! Let's try it:

[<email address hidden>] ~/WORK/jlab $ neutron net-update b09fd0c8-9f81-4a9c-8501-e129843a05d4 --router:external
Updated network: b09fd0c8-9f81-4a9c-8501-e129843a05d4

Looks like it succeeded... but the network still hasn't changed:

[<email address hidden>] ~/WORK/jlab $ neutron net-show b09fd0c8-9f81-4a9c-8501-e129843a05d4
+-------------------------+------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------------------+
| admin_state_up | True |
| contrail:fq_name | default-domain |
| | admin |
| | Public |
| contrail:instance_count | 0 |
| contrail:subnet_ipam | {"subnet_cidr": "10.129.0.0/24", "ipam_fq_name": ["default-domain", "admin", "My-IPAM"]} |
| id | b09fd0c8-9f81-4a9c-8501-e129843a05d4 |
| name | Public |
| shared | False |
| status | ACTIVE |
| subnets | 5da81cd2-32b4-4e95-b957-f2db0e6e72ed |
| tenant_id | 29098dc789f14024be8f83c4e4cc6bc3 |
+-------------------------+------------------------------------------------------------------------------------------+

Well, let's try anyway. As dsmith again, can we view that network...?

[<email address hidden>] ~/WORK/jlab $ neutron net-list
+--------------------------------------+--------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+--------+------------------------------------------------------+
| 59bbc517-86c0-4820-8e5d-dace384370e6 | DREW-0 | 8fbe0727-20f6-4bb0-a574-2565144900ad 192.168.42.0/24 |
| ea29ef39-0ce4-4c44-854e-71bdc1710ae8 | DREW-1 | ae839fac-a829-496f-b240-6c424bb79d92 192.168.23.0/24 |
+--------------------------------------+--------+------------------------------------------------------+

Nope. And trying to allocate an IP from it still fails:

[<email address hidden>] ~/WORK/jlab $ nova floating-ip-create Public
ERROR: FloatingIpPoolNotFound: Floating ip pool not found. (HTTP 404) (Request-ID: req-c627789d-96c6-4c28-a9c5-33332fac2b6d)

[<email address hidden>] ~/WORK/jlab $ nova floating-ip-create public
ERROR: FloatingIpPoolNotFound: Floating ip pool not found. (HTTP 404) (Request-ID: req-8f8236b2-e098-4597-a742-7ef8f88a1ffb)

[<email address hidden>] ~/WORK/jlab $ nova floating-ip-create b09fd0c8-9f81-4a9c-8501-e129843a05d4
ERROR: FloatingIpPoolNotFound: Floating ip pool not found. (HTTP 404) (Request-ID: req-6e262641-5b67-466e-a46a-e312c2df8fd4)

So that's where I'm stuck. Admin can create floating IPs in the public pool, but regular users cannot.