juju upgrade from 1.24-beta2 to 1.24.5 broken

Bug #1493444 reported by Jacek Nykis
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Horacio Durán
1.24
Fix Released
High
Horacio Durán
1.25
Fix Released
High
Horacio Durán

Bug Description

I tried upgrading juju from 1.24-beta2 to 1.24.5 but it got stuck. First it was due to known problem:

https://lists.ubuntu.com/archives/juju/2015-June/005441.html

I worked around it by recreating symlink as per instructions above. This allowed upgrade to get a bit further but eventually upgrade on the bootstrap node but into a loop, showing this every few minutes:

2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: add block device documents for existing machines
2015-09-08 15:51:22 INFO juju.state open.go:51 opening state, mongo addresses: ["127.0.0.1:37017"]; entity machine-0
2015-09-08 15:51:22 DEBUG juju.state open.go:52 dialing mongo
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "172.31.23.113:37017"
2015-09-08 15:51:22 DEBUG juju.state open.go:57 connection established
2015-09-08 15:51:22 DEBUG juju.state open.go:64 mongodb login successful
2015-09-08 15:51:22 INFO juju.state state.go:182 creating lease client as machine-0
2015-09-08 15:51:22 INFO juju.state state.go:194 starting leadership manager
2015-09-08 15:51:22 INFO juju.state state.go:204 starting presence watcher
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:22 DEBUG juju.state open.go:293 closed state without error
2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: add instance id field to IP addresses
2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: migrate charm archives into environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:34 migrating charms to environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/haproxy-28, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/nrpe-external-master-7, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ntp-14, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ubuntu-repository-cache-1, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ubuntu-basenode-1, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ubuntu-repository-cache-2, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/ksplice-3, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/landscape-client-5, already in environment storage
2015-09-08 15:51:22 DEBUG juju.upgrade charmstorage.go:62 skipping local:trusty/haproxy-29, already in environment storage
2015-09-08 15:51:22 INFO juju.upgrade upgrade.go:133 running upgrade step: change entityid field on status history to globalkey
2015-09-08 15:51:22 INFO juju.state open.go:51 opening state, mongo addresses: ["127.0.0.1:37017"]; entity machine-0
2015-09-08 15:51:22 DEBUG juju.state open.go:52 dialing mongo
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:22 DEBUG juju.state open.go:57 connection established
2015-09-08 15:51:22 DEBUG juju.state open.go:64 mongodb login successful
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "172.31.23.113:37017"
2015-09-08 15:51:22 INFO juju.state state.go:182 creating lease client as machine-0
2015-09-08 15:51:22 INFO juju.state state.go:194 starting leadership manager
2015-09-08 15:51:22 INFO juju.state state.go:204 starting presence watcher
2015-09-08 15:51:22 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017"
2015-09-08 15:51:23 DEBUG juju.state open.go:293 closed state without error
2015-09-08 15:51:23 ERROR juju.upgrade upgrade.go:135 upgrade step "change entityid field on status history to globalkey" failed: environment UUID "xxxx": unexpected id: 0
2015-09-08 15:51:23 ERROR juju.cmd.jujud upgrade.go:359 upgrade from 1.24-beta2.1 to 1.24.5 for "machine-0" failed (will retry): change entityid field on status history to globalkey: environment UUID "xxxx": unexpected id: 0

juju status is now showing

  "0":
    agent-state: error
    agent-state-info: 'upgrade to 1.24.5 failed (giving up): change entityid field
      on status history to globalkey: environment UUID "xxxx":
      unexpected id: 0'
    agent-version: 1.24.5

Revision history for this message
Martin Packman (gz) wrote :

Looks like a regression from the fix for bug 1479289.

Changed in juju-core:
importance: Undecided → High
status: New → Triaged
tags: added: status upgrade-juju
Revision history for this message
Martin Packman (gz) wrote :

Manual workaround got us past this:

Get a mongo shell per:

<https://gist.github.com/niedbalski/1306c7810e7c3aee9476>

Then delete all status history entries:

> db.statuseshistory.remove({})

Changed in juju-core:
assignee: nobody → Horacio Durán (hduran-8)
Ian Booth (wallyworld)
Changed in juju-core:
milestone: none → 1.26-alpha1
Changed in juju-core:
status: Triaged → In Progress
Ian Booth (wallyworld)
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.