ERROR upgrade in progress - Juju functionality is limited

Bug #1411502 reported by Ryan Beisner
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Menno Finlay-Smits
1.21
Fix Released
High
Menno Finlay-Smits
1.22
Fix Released
High
Menno Finlay-Smits

Bug Description

In test-driving 1.21beta5 in UOSCI, I see 'ERROR upgrade in progress - Juju functionality is limited.' The same actions succeed with 1.20.14. Also take note that adding 60s sleep between bootstrap and deploy resolved as a temporary workaround.

$ juju bootstrap -e osci-sv01-jdev && juju deploy -e osci-sv01-jdev ubuntu
Bootstrapping environment "osci-sv01-jdev"
Starting new instance for initial state server
Launching instance
 - 9024efb3-0121-4f69-9356-2b8f68e6e3d6
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to 172.18.101.35:22
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/devel/juju-1.21-beta5-trusty-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete
ERROR upgrade in progress - Juju functionality is limited

## waited a bit and charm deploy works:
$ juju deploy -e osci-sv01-jdev ubuntu
Added charm "cs:trusty/ubuntu-0" to the environment.
^[jenkins@juju-beis0-machine-4:~/.juju/ssh$ juju stat -e osci-sv01-jdev
environment: osci-sv01-jdev
machines:
  "0":
    agent-state: started
    agent-version: 1.21-beta5
    dns-name: 172.18.101.35
    instance-id: 9024efb3-0121-4f69-9356-2b8f68e6e3d6
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=2048M root-disk=9216M
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.21-beta5
    dns-name: 172.18.101.36
    instance-id: 415945c3-6dd0-4ddb-8680-c0cab293eb2e
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=2048M root-disk=9216M
services:
  ubuntu:
    charm: cs:trusty/ubuntu-0
    exposed: false
    units:
      ubuntu/0:
        agent-state: started
        agent-version: 1.21-beta5
        machine: "1"
        public-address: 172.18.101.36

## all-machines.log:
http://paste.ubuntu.com/9759763/

## environment info:
juju-core:
  Installed: 1.21-beta5-0ubuntu1~14.04.1~juju1
  Candidate: 1.21-beta5-0ubuntu1~14.04.1~juju1
  Version table:
 *** 1.21-beta5-0ubuntu1~14.04.1~juju1 0
        500 http://ppa.launchpad.net/juju/devel/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     1.20.14-0ubuntu1~14.04.1~juju1 0
        500 http://ppa.launchpad.net/juju/stable/ubuntu/ trusty/main amd64 Packages
     1.20.11-0ubuntu0.14.04.1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
     1.18.1-0ubuntu1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

$ lsb_release -a && uname -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
Linux juju-osci-sv01-jdev-machine-0 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

  osci-sv01-jdev:
    admin-secret: XXXXXXXXXX
    agent-stream: devel
    apt-http-proxy: http://XXXXXXXXXX:8000
    auth-url: http://XXXXXXXXXX:5000/v2.0
    control-bucket: XXXXXXXXXX
    image-stream: daily
    logging-config: <root>=DEBUG
    network: XXXXXXXXXX
    password: XXXXXXXXXX
    region: serverstack
    tenant-name: XXXXXXXXXX
    type: openstack
    use-default-secgroup: true
    use-floating-ip: false
    username: XXXXXXXXXX

Revision history for this message
Ryan Beisner (1chb1n) wrote :
Curtis Hovey (sinzui)
Changed in juju-core:
status: New → Triaged
importance: Undecided → High
milestone: none → 1.23
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

This is most likely due to the same reason as bug #1403738, which was fixed, but not backported for 1.21.

Fix landed with this: https://github.com/juju/juju/pull/1343

I've asked Eric and/or Wayne to attempt the fix this issue by backporting the above PR.

Revision history for this message
Curtis Hovey (sinzui) wrote :

When we reported bug #1403738, we saw a systemic failure in all upgrade tests which included maas-devel-upgrade-trusty-amd64. The tests did pass after the fix arrived. 1.21 never failed. Maybe our mass is configured different from the OIL maas.

Revision history for this message
Eric Snow (ericsnowcurrently) wrote :
Revision history for this message
Eric Snow (ericsnowcurrently) wrote :

That is a backport of the patch that fixed bug #1403738 (for 1.22).

Curtis Hovey (sinzui)
no longer affects: juju-core/1.22
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Thank you all for hopping right on this, appreciate it!

FWIW, I don't think MAAS is a variable here, as we're using the OpenStack provider directly (in UOSCI, not OIL).

Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

I'm afraid the analysis of this issue and the fix aren't correct.

Bug #1403738 was completely 1.22 specific (there is no concept of multi-env collections in 1.21) and the part of that fix which has now been backported to 1.21 (for this ticket) is purely a refactoring to make the code easier to maintain. It has no functional effect.

Here's what I think is actually going on...

When you bootstrap a new environment, Juju creates an environment using the same Juju software version as the client. However, for better or worse, when that environment comes up it will then upgrade to the latest tools in that series (if available). Juju has had this behaviour for a long time. I believe the idea is to ensure that new environments have the latest bug fixes and features available for Juju series in use.

In this case I think the environment was bootstrapped with client in the 1.21 series, but not beta5, and immediately after bootstrap it upgraded to beta5. This is why the "upgrade in progress" error was seen - it's not possible to issue environment changing commands such as "deploy" while the environment is upgrading. Can you please confirm the Juju client version used here?

I can see 2 ways to deal with this:

1. Ensure that automation scripts can handle the "upgrade in progress" error and retry the failed command if that's seen. Upgrades are fairly quick (environment size dependent). A retry every 5-10s would be appropriate.

2. Somehow ensure that the client version is the latest in the series (probably not always practical).

3. Change Juju to not automatically upgrade after bootstrap. This is a significant change would require some discussion on juju-dev.

Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

To be clear, I don't believe that bug #1403738 or bug #1358078 are related to this one.

Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

I've just done some more testing and I can see "upgrade in progress" happening even when the client is 1.21-beta5.

I can see what the issue is: if there's an alpha or beta release tag in the version, upgrade mode is briefly in effect just as the machine agent starts up. No upgrades actually happen though. On my machine, the window where upgrade mode is incorrectly active is the first 4 seconds after the machine agent comes up. That's long enough to be a problem when deployment is automated.

The fix should be easy. Doing this now.

Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

This will probably affect 1.22 and 1.23 as well so updating the ticket to reflect this.

Changed in juju-core:
assignee: nobody → Menno Smits (menno.smits)
Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

Fixed for 1.21 in 2b05f58

Changed in juju-core:
status: Triaged → In Progress
Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

Fixed for 1.22 in dca2be7.
Fixed for 1.23(master) in dbcc0d2.

Changed in juju-core:
status: In Progress → Fix Committed
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Great news, thanks for your work on this. FWIW, I do think that there is a general expectation that when juju bootstrap returns 0, the environment is ready for deployment.

Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Revision history for this message
Ryan Beisner (1chb1n) wrote :

FYI, confirmed resolved with "bootstrap then deploy" automation (juju-core 1.21-rc1-0ubuntu1~14.04.1~juju1).

Also ran through a full OpenStack deployment test. A-OK. Thanks again!

Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

That's great to hear. Thanks for confirming and reporting back.

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.23 → 1.23-beta1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.