autopkgtest-cloud cron emails are low signal, don't tell you if there's a problem or provide info needed to figure out if there is

There have been a lot of emails from the autopkgtest infra over the long (US) holiday weekend, reporting a significant number of dead runners. The emails don't contain enough information to tell why they were killed; e.g:

==== worker <email address hidden> failed ====
[...]
Started autopkgtest worker lcy01-8.
[...]
2018-05-28 03:37:35,394 [26050] WARNING: Testbed failure, retrying in 5 minutes
2018-05-28 03:37:35,394 [26050] ERROR: Three tmpfails in a row, aborting worker. Log follows:
2018-05-28 03:37:35,395 [26050] ERROR: autopkgtest [02:46:11]: git checkout: 5243905 ssh-setup/nova: Add support for keystone v3 auth
autopkgtest [02:46:11]: host juju-prod-ues-proposed-migration-machine-11; command line: /home/ubuntu/autopkgtest/runner/autopkgtest --output-dir /tmp/autopkgtest-work.158kldh_/out --timeout-copy=6000 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-production/setup-canonical.sh --setup-commands /home/ubuntu/autopkgtest/setup-commands/setup-testbed --apt-pocket=proposed=src:graphicsmagick --apt-upgrade diaspora-installer --env=ADT_TEST_TRIGGERS=graphicsmagick/1.3.29+hg15665-1 -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor autopkgtest --security-groups <email address hidden> --name adt-cosmic-amd64-diaspora-installer-20180528-005242 --image adt/ubuntu-cosmic-amd64-server --keyname testbed-juju-prod-ues-proposed-migration-machine-11 --net-id=net_ues_proposed_migration -e ''"'"'http_proxy=http://squid.internal:3128'"'"'' -e ''"'"'https_proxy=http://squid.internal:3128'"'"'' -e ''"'"'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,ppa.launchpad.net'"'"'' --mirror=http://ftpmaster.internal/ubuntu
autopkgtest [02:46:49]: @@@@@@@@@@@@@@@@@@@@ test bed setup
[...]
autopkgtest [02:47:25]: test command1: preparing testbed
Reading package lists...
Building dependency tree...
Reading state information...
Correcting dependencies...Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
 Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
  build-essential cpp cpp-7 dbconfig-common dbconfig-pgsql diaspora-common
  diaspora-installer exim4 exim4-base exim4-config exim4-daemon-light
  fontconfig fontconfig-config fonts-dejavu-core g++ g++-7 gcc gcc-7
  gcc-7-base ghostscript gir1.2-freedesktop gir1.2-gdkpixbuf-2.0
  gir1.2-harfbuzz-0.0 gir1.2-rsvg-2.0 hicolor-icon-theme icu-devtools
  imagemagick imagemagick-6-common imagemagick-6.q16 libasan4 libatomic1
  libavahi-client3 libavahi-common-data libavahi-common3 libbz2-dev libc-ares2
  libc-dev-bin libc6-dev libcairo-gobject2 libcairo-script-interpreter2
  libcairo2 libcairo2-dev libcc1-0 libcilkrts5 libcroco3 libcups2
  libcupsimage2 libcurl4-openssl-dev libdatrie1 libdjvulibre-dev
  libdjvulibre-text libdjvulibre21 libexif-dev libexif12 libexpat1-dev
  libffi-dev libfftw3-double3 libfontconfig1 libfontconfig1-dev
  libfreetype6-dev libgcc-7-dev libgd3 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-common libgdk-pixbuf2.0-dev libglib2.0-bin libglib2.0-dev
  libglib2.0-dev-bin libgmp-dev libgmpxx4ldbl libgomp1 libgraphite2-3
  libgraphite2-dev libgs9 libgs9-common libharfbuzz-dev libharfbuzz-gobject0
  libharfbuzz-icu0 libharfbuzz0b libhttp-parser2.8 libice-dev libice6
  libicu-dev libicu-le-hb-dev libicu-le-hb0 libiculx60 libijs-0.35
  libilmbase-dev libilmbase12 libisl19 libitm1 libjbig-dev libjbig0
  libjbig2dec0 libjemalloc1 libjpeg-dev libjpeg-turbo8 libjpeg-turbo8-dev
  libjpeg8 libjpeg8-dev liblcms2-2 liblcms2-dev liblqr-1-0 liblqr-1-0-dev
  liblsan0 libltdl-dev libltdl7 liblzma-dev libmagickcore-6-arch-config
  libmagickcore-6-headers libmagickcore-6.q16-3 libmagickcore-6.q16-3-extra
  libmagickcore-6.q16-dev libmagickwand-6-headers libmagickwand-6.q16-3
  libmagickwand-6.q16-dev libmagickwand-dev libmpc3 libmpx2
  libnginx-mod-http-geoip libnginx-mod-http-image-filter
  libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream
  libopenexr-dev libopenexr22 libpango-1.0-0 libpangocairo-1.0-0
  libpangoft2-1.0-0 libpaper1 libpcre16-3 libpcre3-dev libpcre32-3
  libpcrecpp0v5 libpixman-1-0 libpixman-1-dev libpng-dev libpq-dev libpq5
  libpthread-stubs0-dev libquadmath0 librsvg2-2 librsvg2-common librsvg2-dev
  libruby2.5 libsm-dev libsm6 libssl-dev libstdc++-7-dev libthai-data libthai0
  libtiff-dev libtiff5 libtiff5-dev libtiffxx5 libtsan0 libubsan0 libuv1
  libwebp6 libwmf-dev libwmf0.2-7 libx11-dev libxau-dev libxcb-render0
  libxcb-render0-dev libxcb-shm0 libxcb-shm0-dev libxcb1-dev libxdmcp-dev
  libxext-dev libxml2-dev libxpm4 libxrender-dev libxrender1 libxslt1-dev
  libxt-dev libxt6 linux-libc-dev nginx nginx-common nginx-core nodejs
  pkg-config poppler-data postgresql postgresql-10 postgresql-client
  postgresql-client-10 postgresql-client-common postgresql-common
  python3-distutils python3-lib2to3 rake redis-server redis-tools ruby
  ruby-dev ruby-did-you-mean ruby-diff-lcs ruby-minitest ruby-net-telnet
  ruby-power-assert ruby-rspec ruby-rspec-core ruby-rspec-expectations
  ruby-rspec-mocks ruby-rspec-support ruby-test-unit ruby-thread-order ruby2.5
Deleting existing group {'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'name': '<email address hidden>', 'description': 'copy <email address hidden> of default (Default security group)', 'security_group_rules': [{'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'description': '', 'ethertype': 'IPv4', 'id': '15657ed8-c781-4080-9677-8ede743e30cd', 'port_range_min': None, 'created_at': '2018-05-27T09:20:17Z', 'remote_group_id': None, 'security_group_id': '5c87742c-aaa3-4806-932e-9317521587dc', 'revision_number': 1, 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'port_range_max': None, 'protocol': 'icmp', 'remote_ip_prefix': '91.189.90.53/32', 'updated_at': '2018-05-27T09:20:17Z', 'direction': 'ingress'}, {'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'description': '', 'ethertype': 'IPv4', 'id': '3c9ef2c0-8618-4e2c-b97c-b43ce7068a9e', 'port_range_min': None, 'created_at': '2018-05-27T09:20:18Z', 'remote_group_id': '843e3a73-f1ee-4ecc-bc8c-b2d297df96b9', 'security_group_id': '5c87742c-aaa3-4806-932e-9317521587dc', 'revision_number': 1, 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'port_range_max': None, 'protocol': None, 'remote_ip_prefix': None, 'updated_at': '2018-05-27T09:20:18Z', 'direction': 'ingress'}, {'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'description': '', 'ethertype': 'IPv4', 'id': '6f392f68-54ca-4c0b-a5ba-daa7a3999d6a', 'port_range_min': 22, 'created_at': '2018-05-27T09:20:17Z', 'remote_group_id': None, 'security_group_id': '5c87742c-aaa3-4806-932e-9317521587dc', 'revision_number': 1, 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'port_range_max': 22, 'protocol': 'tcp', 'remote_ip_prefix': '162.213.33.179/32', 'updated_at': '2018-05-27T09:20:17Z', 'direction': 'ingress'}, {'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'description': '', 'ethertype': 'IPv4', 'id': '7ac1770e-e6ce-4843-b114-e4f24281f784', 'port_range_min': 22, 'created_at': '2018-05-27T09:20:18Z', 'remote_group_id': None, 'security_group_id': '5c87742c-aaa3-4806-932e-9317521587dc', 'revision_number'
: 1, 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'port_range_max': 22, 'protocol': 'tcp', 'remote_ip_prefix': '91.189.90.53/32', 'updated_at': '2018-05-27T09:20:18Z', 'direction': 'ingress'}, {'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'description': '', 'ethertype': 'IPv6', 'id': '7bf03f0d-432b-4c46-b2b1-c12af62793f7', 'port_range_min': None, 'created_at': '2018-05-27T09:20:17Z', 'remote_group_id': '843e3a73-f1ee-4ecc-bc8c-b2d297df96b9', 'security_group_id': '5c87742c-aaa3-4806-932e-9317521587dc', 'revision_number': 1, 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'port_range_max': None, 'protocol': None, 'remote_ip_prefix': None, 'updated_at': '2018-05-27T09:20:17Z', 'direction': 'ingress'}, {'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'description': '', 'ethertype': 'IPv6', 'id': 'ec28f1fe-75cf-49f8-9e2c-e83afc72da24', 'port_range_min': None, 'created_at': '2018-05-27T09:20:18Z', 'remote_group_id': None, 'security_group_id': '5c87742c-aaa3-4806-932e-9317521587dc', 'revision_number': 1, 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'port_range_max': None, 'protocol': None, 'remote_ip_prefix': None, 'updated_at': '2018-05-27T09:20:18Z', 'direction': 'egress'}, {'tenant_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'description': '', 'ethertype': 'IPv4', 'id': 'f6b2100e-f068-4111-9f68-6e86e6250b29', 'port_range_min': None, 'created_at': '2018-05-27T09:20:18Z', 'remote_group_id': None, 'security_group_id': '5c87742c-aaa3-4806-932e-9317521587dc', 'revision_number': 1, 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'port_range_max': None, 'protocol': None, 'remote_ip_prefix': None, 'updated_at': '2018-05-27T09:20:18Z', 'direction': 'egress'}], 'id': '5c87742c-aaa3-4806-932e-9317521587dc', 'updated_at': '2018-05-27T09:20:18Z', 'created_at': '2018-05-27T09:20:17Z', 'project_id': 'afaef86b96dd4828a1ed5ee395ea1421', 'revision_number': 10}
<fin>

The only verbose openstack output here is about deletion of a security group. Nothing in this log directly links back to a particular openstack instance, or gives any hint of why the instance has disappeared. The comments in the source suggest this is due to hitting quotas when trying to run too many huge jobs in parallel; but that should not be a problem with proper quotas, and doesn't explain why there appears to be a recent increase, and none of the output corroborates that this is what is happening.

If we have frequently failing autopkgtest instances, we ought to know why. For that, we need better logging.

If the failures are not interesting, then we should not generate emails for all of them; we should detect that they're not interesting, and avoid sending mail.