Upload of 1.5MB File Caused System to Hang

Assigning to Shane. Not sure what the next actionable step might be, perhaps we connect with Anthony to recreate. Perhaps we write a load test suite.

I have reconfirmed this today. I tried to attach a large PDF to a blog entry and the system hangs every time. Interestingly, if you attach the same file in the comments section it works as expected. I'm attaching the file I used so you can replicate it.

Very good, thanks Nat!

--Paul

On Apr 28, 2009, at 1:24 PM, Nat Katin-Borland wrote:

> I have reconfirmed this today. I tried to attach a large PDF to a
> blog
> entry and the system hangs every time. Interestingly, if you attach
> the
> same file in the comments section it works as expected. I'm attaching
> the file I used so you can replicate it.
>
> ** Attachment added: "CPallaris_New_School_Briefing.pdf"
> http://launchpadlibrarian.net/26085006/CPallaris_New_School_Briefing.pdf
>
> --
> Upload of 1.5MB File Caused System to Hang
> https://bugs.launchpad.net/bugs/366359
> You received this bug notification because you are subscribed to
> KARL3.
>
> Status in Porting KARL to a new architecture: New
>
> Bug description:
> Upload of 1.5MB File Caused System to Hang

Thanks Nat, now we can hopefully make progress.

It appears that Nat's 5 MB PDF is revealing a different problem. It does not actually cause the server to hang. The PDF exceeds the upload limit, so the server returns an HTTP 400 bad request error. Browsers handle this response by showing either an error message or a blank page, which is a terrible user experience that makes the server look broken, but the server continues to function.

I will open a separate bug for solving the UI problem.

With the UI problems fixed, we need to resume the hunt for the cause of the lockup we saw on April 24.

The system appears to be hung again. On the first attempt, the file upload failed and hung, but the system was usable. Opening a second window and uploading the same file, hung the system. Now, we are out of the water and the site seems to be unavailable. I am attaching the file that I have attempted to upload.

Ah-ha! wvWare is doing this. I used gdb to find out what Python is doing while it's not responding. It turns out it's waiting for a response from a subprocess running wvWare, a document conversion program. The problem is that wvWare is and always has been slightly unstable, and with Anthony's document, wvWare simply hangs forever, not even consuming CPU time.

karl.utilities.converters.baseconverter calls wvWare. That code needs a timeout that kills wvWare and fails the attempted upload.

This turned out simpler than I expected. We were using a method of the subprocess module that is documented to deadlock sometimes. By switching from the wait() call to the communicate() call, the problem went away. So wvWare is not implicated after all.