Something causes intermittent, daily, non-scheduled 502s, visible in webservice

As described in bug 380504, webservice users report that requests return 502s intermittently. They said that this happened on production, daily, at different times throughout the day.

This seems to rule out deployment as the only cause (see bug 640065); it also seems to rule out at least some of our own scheduled application state changes, such as sending signals for log rotation.

The original bug was closed when lazr.restfulclient got the ability to retry. However, the underlying problem in Launchpad presumably still exists.

In https://bugs.launchpad.net/lazr.restfulclient/+bug/380504/comments/31 , I described a hypothesis from Francis Lacoste that HAProxy might be the cause. At the time, we were blocked from investigating because the LOSAs did not have access to the logs. Since then, LOSAs have gained access at our request, though they still cannot give us direct access because of operational concerns. We should take advantage of this and pursue that avenue.

Note also bug 636713; Francis' hypothesis is essentially that a sustained overload will also result in the symptoms of this bug.