Enblend fails to recognise output tiff file too large.

You have mingled two issues in one report.

(1) Enblend does not warn early, if the output image
will overwhelm the selected output format.

(2) Neither Enblend nor Enfuse clean up a half-done
output image, if they unexpectedly die or they are
interrupted by the user.

Issue (1): In the general case this is a complicated
problem that requires (a) finding out about the
final image size as early in the process as possible
and (b) knowing the size limits of all possible output
image formats with respect to their actual libraries.

Issue (2) should be fixed in rev236fd8c3e767.

Let's consider the "issue 2" as "fix released" and continue this discussion on "issue 1" as you call it.

restrictions that I know of right now, is 4Gb of pixel data for TIFF files and 64k pixels per side for JPG.

How about if we make this a warning.

if (((height * width * estimate_for_bytes_per_pixel) > 2LL^32)) &&
     (outputformat == TIFF)) {
    ..... WARNING: possibly generating more than 4Gb of pixel data. May not fit in output TIFF file.
}
if (((width > 65500) ||
     (height > 65500)) &&
    (outputformat == JPG)) {
   ..... WARNING: JPG format may not support the output image size.
}

Then it isn't as critical, and doesn't have to be 100% right. We can easily add checks for limits on other output formats as we go along.