Juju fails to bootstrap if memory is lower than 1GB

So there are 2 bugs here

1) This used to be working. I bootstrapped 2 days ago with the same code, and everything worked
2) We think we have a "minMemoryHeuristic" which requests instances that have at least 1GB of RAM by default. But the default is actually a 512MB instance (it was 2 days ago, but it was working).

The suspicion here is that a recent apt-xapian security update, that post-dates the current images, is pulled in on the apt_upgrade step of cloud init, which triggers a cpu and memory intensive reindexing during the remainder of boot. This pushes us over the memory limit, not just for bootstrap, but also for some early hooks on memory sensitive charms like mysql.

Bug 1227425 covers removing apt-xapian-index from cloud-images entirely, which seems like the best solution,.

I tried a hackaround of purging the package in a cloud-config script, but this can only happen after apt_upgrade has been run, and the package is not smart enough to stop the ongoing indexing, instead it just blows up at a later point when it realises its files have disappeared:

Traceback (most recent call last):
  File "/usr/sbin/update-apt-xapian-index", line 108, in <module>
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 735, in rebuild
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 715, in buildIndex
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 64, in finished
  File "/usr/share/apt-xapian-index/plugins/cataloged_time.py", line 126, in finished
IOError: [Errno 2] No such file or directory: '/var/lib/apt-xapian-index/cataloged_times.p.new'
Exception OSError: (2, 'No such file or directory', '/var/lib/apt-xapian-index/update-socket') in <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress instance at 0x7fca8afe4f80>> ignored

Note, but per bug Bug 1227425 the recent cloud images should resolve this.

-Thanks,
Antonio

I can confirm that I can "juju bootstrap" to a 512MB machine on Canonistack. So I think the reindexing was, indeed, the cause of why it would fail with not enough memory.

We still have the bug that we are trying to bootstrap with <1GB by default, even though we *have* a heuristic that we should default to 1GB if not specified.

This was actually fixed in between 1.17.0 and 1.17.2 We are bootstrapping on 512M again.