NovaLogger.exception() should not dump a stack trace unless --verbose is specified

This is probably low priority or wishlist, but it's still a nice little task for me to pick off :)

Suggestion:

You can probably just modify handle_exception in log.py

Vish

On Feb 25, 2011, at 4:05 PM, Kevin L. Mitchell wrote:

> This is probably low priority or wishlist, but it's still a nice little
> task for me to pick off :)
>
> ** Changed in: nova
> Assignee: (unassigned) => Kevin L. Mitchell (klmitch)
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/725349
>
> Title:
> NovaLogger.exception() should not dump a stack trace unless --verbose
> is specified
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> We should only be verbose if we're requested to be verbose :)

2011/2/26 Kevin L. Mitchell <email address hidden>:
> We should only be verbose if we're requested to be verbose :)

A stack trace doesn't sound like debug info to me. It sounds like
something I'd output with LOG.error, so this would never get filtered.

Do you have an example of a stack trace you feel is just noise that
should be filtered out?

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

A stack trace is debugging information, pure and simple. Users (e.g., Nova admins) as a rule don't have the tools or knowledge to interpret the stack trace. Now, for most exceptions, obviously, the thing to do is email the stack trace to the developers. However, there are some exceptions which are not errors the developers can do anything about--things like not being able to reach the database server, for instance, which I can't figure out a way to do without violating abstraction boundaries. In those cases, the stack traces are noise for *both* developers *and* users.

The idea here is that, in the normal case where a user has set up Nova, the log files contain only the actual exception information, minus stack traces. That exception information can be useful to the user ("Hmmm, can't reach my database server, I wonder why?"). If they run across an actual Nova bug, on the other hand, they re-start the appropriate Nova service with --verbose, replicate the bug, then attach the stack trace.

This idea/bug, by the way, came out of a conversation vishy and I had regarding stack traces in the case that the AMQP server could not be reached. In that case, it was easy to intercept the exception without violating abstraction boundaries (in fact, the exception was already being caught and the connection was being retried), but it was almost impossible to determine, from the log, what was happening because of the noise from the stack traces. I made the same point to him ("stack traces are for developers!"), and he made a comment to the effect that perhaps we could suppress stack traces unless --verbose was specified. As it turns out, this was the only way I could see to fix a similar noise issue related to the database server being unreachable.

2011/3/7 Kevin L. Mitchell <email address hidden>:
> The idea here is that, in the normal case where a user has set up
> Nova, the log files contain only the actual exception information,
> minus stack traces.  That exception information can be useful to the
> user ("Hmmm, can't reach my database server, I wonder why?").  If they
> run across an actual Nova bug, on the other hand, they re-start the
> appropriate Nova service with --verbose, replicate the bug, then
> attach the stack trace.

If all you get in the log file is "An exception occurred" or perhaps "A
KeyError exception occurred", I seriously doubt you'll be able to guess
what triggered it so that you can reproduce it. Even with the stack
trace, it can be hard enough.

Can you give me an example of a stack trace that you feel should be
removed and what exactly you'd log instead?

I actually think it's extremely useful to be able to leave out --verbose
to avoid the debug information when everything is working fine, but
still get the stack trace if something actually goes wrong.

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

On Mon, 2011-03-07 at 22:08 +0000, Soren Hansen wrote:
> If all you get in the log file is "An exception occurred" or perhaps "A
> KeyError exception occurred", I seriously doubt you'll be able to guess
> what triggered it so that you can reproduce it. Even with the stack
> trace, it can be hard enough.

I never said you only get "An exception occurred". The exception is
still printed out in its entirety; the only thing that's omitted is all
the lines that begin with "TRACE:" (or whatever you've configured). In
fact, the exception information ends up being printed twice,
currently--once as part of the ERROR message, then again at the end of
the TRACE, and its rendered identically in both places.

> Can you give me an example of a stack trace that you feel should be
> removed and what exactly you'd log instead?

See bug 715620. I cannot find a way to catch that exception without
either violating an abstraction boundary (trying to catch the specific
DB exception which itself isn't very specific) or casting far too wide a
net (catching a whole bunch of exceptions, some of which indicate
configuration problems for which a trace is noise, and some of which
could indicate software errors, for which a trace might be useful). My
patch corrects the issue, in that it prints out the exception message;
if the SQL connection can't be reached, that is easy to see.

> I actually think it's extremely useful to be able to leave out --verbose
> to avoid the debug information when everything is working fine, but
> still get the stack trace if something actually goes wrong.

To my mind, stack traces *are* debug information. We could add
--stacktraces/--nostacktraces, I suppose, but it would have to default
to being off...
--
Kevin L. Mitchell <email address hidden>