Tests failing with no explanatory text - Need a runtime log from c-c-s to debug things.

Looking at the submission file, the following are missing any output:

memory/check
- Failed in results. Nothing in comments. looking at the script, it prints some data to stdout and some to stderr.

virtualization/kvm_check_vm
- Failed in results. Nothing in comments. Also prints to both stdout and stderr.

memory/info
- Failed in results. Nothing in comments. Always returns 0... should not have failed... Prints only to stdout.

I think to start with, at this point, getting a traceable log from the test tool is critical. Without being able to look back at the run as it was happening, there's no way to debug this.

Hi Rod,

Could you please install the plainbox package, then run this? this will run just memory/info:

plainbox run -D -C -i 2013.com.canonical.certification::memory/info -f xml -o memory-info.xml

using -D -C will output insane amounts of text, you can ignore them as they will be logged to file for later analysis.

I'd be interested in seeing both the resulting xml file (should be short) and the (long) log files which will be produced in ~/.cache/plainbox/logs.

For the record, I tried reproducing this behavior using source from trunk and it doesn't happen; I see all the output even if a test fails. This is not identical to your installation though, so *I* may be doing something wrong. I'll keep trying to get this behavior too.

Daniel, I'm currently re-running the whole test suite on the C220, since the experience with the Intel suggests it may succeed this time around. This system takes a long time to finish its disk tests, so it may be tomorrow before I'll be able to run the plainbox test you requested; but see below....

I've run into this problem on a third system, a SeaMicro node. I'm attaching the output files. Note that a different set of tests failed and produced null results this time. The memory tests all seemed OK on this system. Nonetheless, I did try your plainbox test, but it doesn't seem to like the command you gave me:

$ plainbox run -D -C -i 2013.com.canonical.certification::memory/info -f xml -o memory-info.xml
usage: plainbox [-h] [--version] [--providers {all,stub}] [-v] [-D] [-C]
                [-T LOGGER] [-P] [-I]
                {run,self-test,check-config,dev,startprovider} ...
plainbox: error: unrecognized arguments: -D -C

The plainbox package installed is 0.6~dev+bzr2905+pkg2~ubuntu14.04.1.

More failures. Attached are logs from two SeaMicro nodes. One of these (in the node-63-0 directory in the tarball) is the same node reported earlier. Note that the pattern of blank failures is different for this run than from the previous one. The node-50-0 directory holds the results from a run on another node on the same SeaMicro box. Neither of these failed on the specific test whose output was requested earlier. Was there a reason you wanted that specific test, or was it just any failed test? If the latter, I could try with a test that fails -- but as these results indicate, that's a moving target.

Hi Rod,

Apologies, I fumbled the plainbox command, and -D -C have to come *before* run:

plainbox -D -C run -i 2013.com.canonical.certification::memory/info -f xml -o memory-info.xml

I'm just choosing this test because I see it failed, but this problem seems to be intermittent. memory/info is relatively quick to run though, and it's been known to fail on occasion, so (I hope this is not too onerous for you) could you run it until you get it to not attach output in the xml/html, and then attach the log files?

FWIW I did a run on a VM, I chose some of the tests that had the problem in your submission, but that are quick to run (rtc, storage-preinserted with no usb stick, ipmi on a system with no ipmi support), and all of them did include command output in the result. I may have just lucked out though... I'll keep looking at this.

Daniel,

I've tried running that memory/info test dozens of times in isolation with no failures. Therefore, I don't think the problem lies in the individual commands but in the c-c-s framework or the way the commands are interacting with each other, at least on some computers. (I've only observed this problem on a couple of Ciscos and a SeaMicro.) Would it help if you had access to one of the systems that produces failures?

We are also seeing this at IBM across machine types. I'm attaching results files for the same HW, that were run before and after this problem showed up.