Comment 25 for bug 1491688

I've been digging further into this. I now understand what causes the mongodb "blip". When the replicaset config is changed in any significant way, the MongoDB instance temporarily drops from PRIMARY to SECONDARY, even when there's just one node.

I could reproduce the blip by introducing a machine agent worker like this:

    a.startWorkerAfterUpgrade(runner, "breaker", func() (worker.Worker, error) {
     return worker.NewSimpleWorker(func(<-chan struct{}) error {
      logger.Infof("breaker: getting current replicaset config")
      config, err := replicaset.CurrentConfig(st.MongoSession())
      if err != nil {
       return err
      }
      n := len(config.Members)
      if n != 1 {
       return errors.Errorf("breaker: expected 1 member, saw %d", n)
      }
      m := config.Members[0]
      v := float64(2)
      m.Priority = &v
      config.Members[0] = m
      logger.Infof("breaker: setting new replicaset config")

      <-rsyslog.BreakCh
      close(rsyslog.BreakCh)

      return replicaset.Set(st.MongoSession(), config.Members)
     }), nil
    })

I've been investigating all sorts of possibilities in terms of if and how the blip might have contributed to the rsyslog issue. I feel like I might be close but I'm not there yet.