maas provider: 1.21b3 removes ip from api-endpoints

The `api-info` command was added to be able to inspect the information that the juju command line client was using to connect to the API server. `api-info` shows all the information.

`api-endpoints` says `purpose: Print the API server address`. The `api-endpoints` has only ever returned one address and will continue to only return one address. The address it returns is the most recent address used to connect to the API server.

Every time the API connects, the list that is saved in the .JENV file is updated. The endpoint that you are talking to is always the first one.

There is no regression here, and it is unclear if there is even a problem.

If you are having problems, can you please explain the problem?

@thumper -- you are right about the number of results returned. I think I was mistaking all the state servers vs. api-endpoints. There is still a regression here.

juju 1.20.13:

dpb@helo:trunk$ juju bootstrap -v
Launching instance
WARNING picked arbitrary tools &{1.20.13-trusty-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.20.13-trusty-amd64.tgz 85cbca755510844e6345704a91f3b5bb308edb0d2b6e3ab66bfe847b88edf28b 8120371}
 - /MAAS/api/1.0/nodes/node-2db80c2e-4b4a-11e4-ad24-a0b3cce4ecca/
Waiting for address
Attempting to connect to maxwell.beretstack:22
Attempting to connect to maxwell.beretstack:22
Attempting to connect to 10.1.4.2:22
Warning: Permanently added '10.1.4.2' (ECDSA) to the list of known hosts.
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: git
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz 'https://streams.canonical.com/juju/tools/releases/juju-1.20.13-trusty-amd64.tgz'
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
dpb@helo:trunk$ juju api-endpoints
10.1.4.2:17070

juju 1.21beta3:

dpb@helo:trunk$ juju bootstrap -v
Bootstrapping environment "virtue"
Starting new instance for initial state server
Launching instance
WARNING no architecture was specified, acquiring an arbitrary node
 - /MAAS/api/1.0/nodes/node-2db80c2e-4b4a-11e4-ad24-a0b3cce4ecca/
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to maxwell.beretstack:22
Attempting to connect to maxwell.beretstack:22
Attempting to connect to 10.1.4.2:22
Warning: Permanently added '10.1.4.2' (ECDSA) to the list of known hosts.
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/devel/juju-1.21-beta3-trusty-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete
dpb@helo:trunk$ juju api-endpoints
maxwell.beretstack:17070

After this was done, I did a "--refresh" and I got back the ip address. So something appears to be changed in the way the initial data is put out.

On 1 December 2014 at 20:45, Tim Penhey <email address hidden> wrote:

> The `api-info` command was added to be able to inspect the information
> that the juju command line client was using to connect to the API
> server. `api-info` shows all the information.
>
> `api-endpoints` says `purpose: Print the API server address`. The `api-
> endpoints` has only ever returned one address and will continue to only
> return one address. The address it returns is the most recent address
> used to connect to the API server.
>

Wish that were true. Juju 1.19.3 (at least) returned multiple addresses
from this command. Note the plurality of the command itself too suggests
that multiple addresses could be returned. Whilst I have no complaint with
a single address being returned, it certainly is a regression as David
shows.

@Adam -- I have seen it return multiple addresses as well, just couldn't
find a version that did it! Thanks for chiming in. Yes, I also find it
odd that the 'help' was used as defense for this, yet the command itself is
plural.

On Tue, Dec 2, 2014 at 3:12 AM, Adam Collard <email address hidden>
wrote:

> On 1 December 2014 at 20:45, Tim Penhey <email address hidden>
> wrote:
>
> > The `api-info` command was added to be able to inspect the information
> > that the juju command line client was using to connect to the API
> > server. `api-info` shows all the information.
> >
> > `api-endpoints` says `purpose: Print the API server address`. The `api-
> > endpoints` has only ever returned one address and will continue to only
> > return one address. The address it returns is the most recent address
> > used to connect to the API server.
> >
>
> Wish that were true. Juju 1.19.3 (at least) returned multiple addresses
> from this command. Note the plurality of the command itself too suggests
> that multiple addresses could be returned. Whilst I have no complaint with
> a single address being returned, it certainly is a regression as David
> shows.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1397376
>
> Title:
> maas provider: 1.21b3 removes ip from api-endpoints
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1397376/+subscriptions
>

--
David Britton <email address hidden>

To summarise the behaviour change: 1.21+ is more likely to get a dns name back, but the users of this command need an ip address. Scripts that depend on the ip address will need to be changed to call --refresh to get the expected behaviour of older jujus.

We need to know if tools are built on this unspecified behaviour. This is not a regression. Juju is now more likely to return the dns address and --refresh can address that, thoug api-info may be more efficient at selected the kind of address desired. If crucial tools will fail if they get an DNS address, we need to consider if this feature can be changed. If anything depends on getting a DNS address, then no chanch can be made.

juju-deployer breaks with this change, which is how I discovered the problem (it's how our official instructions say to deploy the openstack autopilot).

The 'openstack-installer' command from the 'openstack' ubuntu package calls juju-deployer directly, AIUI.

OpenStack autopilot is broken by the change in behaviour. 1.21.x api-endpoints must return an ip address as reliably as 1.20.x. the reality is most institutions will not have MAAS doing DNS.

So options...
A) We change api-endpoints to return an IP address
B) We change api-endpoints to return an IP address for MAAS
C) we return multiple address...but I think this violates 1.18 compatibility. The command did return multiple address during a beta period, but it was reverted (possibly because of it was a compatability issue.)

latest versions of deployer (0.4.x) try all addresses stored in the jenv
file in turn with short time outs, obviously one of them has to work else
the cli is broken. i agree though that juju enforcing go try all at once in
parallel is kinda of painful for any other language and juju should promote
index 0 for state server be 'public' ip address for the server. also as
noted on previous bugs about api-endpoints.. multiple state servers be
broken out instead of a flat list is useful to clients that actually want
to use multiple state servers.. rather than parallel connect to grab bag of
addresses of which only a few would ever work.

On Tue, Dec 2, 2014 at 11:24 AM, David Britton <email address hidden>
wrote:

> juju-deployer breaks with this change, which is how I discovered the
> problem (it's how our official instructions say to deploy the openstack
> autopilot).
>
> The 'openstack-installer' command from the 'openstack' ubuntu package
> calls juju-deployer directly, AIUI.
>
> ** Changed in: juju-core
> Status: Incomplete => New
>
> ** Changed in: juju-core/1.21
> Status: Incomplete => New
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> https://bugs.launchpad.net/bugs/1397376
>
> Title:
> maas provider: 1.21b3 removes ip from api-endpoints
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1397376/+subscriptions
>

The change happened due to the way we sort addresses (HostPorts) now, preferring hostnames to IPs if possible with the assumption they're more likely to be usable. Clearly this is not good enough, especially for MAAS.

I propose to fix the issue taking the following assumptions in mind:
1. api-endpoints *by default* will return a single IP address + port
2. api-info works like now, except the "state-servers" section is always sorted so that the first endpoints is *always* and IP + port
3. api-endpoints --all returns all known endpoints, except unusable ones (machine-local and link-local; like now), but the first endpoints is always an IP + port
3. If we only have a hostname as endpoint, try to resolve it locally before returning it from api-endpoints / api-info, and when saving it into the .jenv file.

This can be implemented in trunk and backported to 1.21 and 1.20.
@David, @Kapil, @Adam, @Tim: Will this work for everyone involved?

On 3 December 2014 at 15:58, Dimiter Naydenov <
<email address hidden>> wrote:

> I propose to fix the issue taking the following assumptions in mind:
> 1. api-endpoints *by default* will return a single IP address + port
> 2. api-info works like now, except the "state-servers" section is always
> sorted so that the first endpoints is *always* and IP + port
> 3. api-endpoints --all returns all known endpoints, except unusable ones
> (machine-local and link-local; like now), but the first endpoints is always
> an IP + port
> 3. If we only have a hostname as endpoint, try to resolve it locally
> before returning it from api-endpoints / api-info, and when saving it into
> the .jenv file.
>
> This can be implemented in trunk and backported to 1.21 and 1.20.
> @David, @Kapil, @Adam, @Tim: Will this work for everyone involved?
>

Sounds great. +1

@Dimiter, I'm not sure we need to bother with api-info. That command just gets the information out of the JENV file. If we change how we store things in the JENV file, then api-info will reflect that.

I believe the bigger change is for api-endpoints and returning a usable host:port pair.

I'm unclear on the option 3 and what benefit it would give.

@Tim, the changes to api-info will come as a result of changing the default sort order for addresses (hostnames orderer after public IPs). As for the 3rd option - it's a fallback in case we have only a hostname as endpoint.

Ok, it seems the proposed solution works for those affected, I'm starting to work on the fix.

Fixing this took a lot more than expected and needed lots of live testing to ensure no regressions.

Proposed with https://github.com/juju/juju/pull/1297 (for 1.22, backport for 1.21 in progress, but shouldn't be long).

Fix committed on trunk, working on a backport for 1.21 as well.