<stub> mthaddon: Do you have the log from building standalone_1 ? It isn't complete - it is missing the authdb replication set.
<mthaddon> stub: https://pastebin.canonical.com/26363/ - gah, was sitting in my screen session - will make sure to put the logs in a consistent place next time
<mthaddon> stub: should I run security.py again?
<stub> mthaddon: I'll worry about it next week.
<mthaddon> stub: I could just run it again now?
<stub> mthaddon: That still doesn't explain why the script only subscribed one replication set instead of both
<mthaddon> stub: not sure I understand? surely this only wants one replication set (the other is auth stuff that doesn't go here?)
<stub> Launchpad needs the authdb and the lpmain replication sets. email addresses are in the authdb set for example.
<mthaddon> but when we're in read-only we're still pointing to the authdb set
<mthaddon> have re-run the security.py on standalone db - seemed okay this time
<stub> mthaddon: Launchpad requires that the authdb replication set exists along side its lpmain replication set, because it issues queries that join in tables in both sets.
<stub> ie. Launchpad will issue 'SELECT Person.name from Person, EmailAddress WHERE Person.id = EmailAddress.person and EmailAddress.email LIKE 'stuart%' ' and expect it to work. At the moment, on the standalone db it will fail because there is no data in the EmailAddress table.
<mthaddon> stub: so does that mean new-slave.py is broken and we need to throw away launchpad_standalone_1 and rebuild it once that's fixed?
<stub> mthaddon: To alleviate the load issues, I've reshaped the replication cluster for the lpmain replication set. Previously, it was a start where everynode was subscribed directly to launchpad_prod_3. Now, only launchpad_prod_1 is subscribed directly, launchpad_prod_2 and launchpad_prod_4 are subscribed via launchpad_prod_1 and launchpad_standalone_1 is subscribed via launchpad_prod_2.
<stub> mthaddon: I need to look at the script. It might be that the authdb subscription happens after the security.py run, which means the script is fine and I can repair it.
<mthaddon> stub: we run security.py as a separate step after ./new-slave.py
<stub> mthaddon: If new-slave.py needs fixing, then we need to rebuild it to test the fixes.
<stub> mthaddon: Hmm... bugger.
Marking critical because we need to be able to rebuild the slave between each rollout otherwise read-only mode won't work.