test run don't match test installed in kernel CI jobs

Bug #1034483 reported by Alexander Sack
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
LAVA Dispatcher
Fix Released
Medium
Andy Doan
lava-kernel-ci-views
Fix Released
Undecided
Unassigned

Bug Description

guess symptom of us not getting back proper failure reports if something goes wrong ... anyway, looking at the snowball kernel CI view seems that random "run" results get not reported, while more tests are getting installed.

http://validation.linaro.org/lava-server/kernel-ci-views/per_board/snowball_sd

pandaboard feels a bit better:
http://validation.linaro.org/lava-server/kernel-ci-views/per_board/panda

In what cases don't we see results? How can this problem be addressed?

Related branches

Revision history for this message
Andy Doan (doanac) wrote :

looking at:

 http://validation.linaro.org/lava-server/scheduler/job/27529/log_file

The problem appears to be that we were unable to get the test results from the target device (see end of log file). At least one of the bundles we pulled didn't have valid JSON for some reason. Here's the guilty snippet of code:

                try:
                    tar = tarfile.open(result_path)
                    for tarinfo in tar:
                        if os.path.splitext(tarinfo.name)[1] == ".bundle":
                            f = tar.extractfile(tarinfo)
                            content = f.read()
                            f.close()
                            device_bundles.append(json.loads(content))
                    tar.close()
                except:
                    logging.warning(traceback.format_exc())
                    status = 'fail'
                    err_msg = err_msg + " Some test case result appending failed."
                    logging.warning(err_msg)

I think we could improve this code in a couple of ways:

1) put exception handling in the "for tarinfo in tar" loop so that the failure of handling one bad bundle doesn't cause of us to not parse all bundles

2) log the actual bundle that failed to parse (and maybe dump its contents)?

Revision history for this message
Andy Doan (doanac) wrote :

I just had another idea to make this fix better. In the event of an exception that's happening while doing the json.loads, we could add that content as an attachment to the "test_results" bundle so that we could analyze what the actual failure was.

I'll be doing some minor dispatcher fixes this week and will try to slip this in with those.

Changed in lava-dispatcher:
importance: Undecided → Medium
assignee: nobody → Andy Doan (doanac)
milestone: none → 2012.08
Fathi Boudra (fboudra)
Changed in lava-dispatcher:
status: New → In Progress
milestone: 2012.08 → 2012.09
Andy Doan (doanac)
Changed in lava-dispatcher:
milestone: 2012.09 → 2012.10
Fathi Boudra (fboudra)
Changed in lava-dispatcher:
milestone: 2012.10 → 2012.11
Andy Doan (doanac)
Changed in lava-dispatcher:
status: In Progress → Fix Released
Changed in lava-kernel-ci-views:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.