Comment 6 for bug 1103343

So really, the root problem lies here:

try:
            # Let's dump the output into file as it can be very large
            # and we don't want to store it in memory
            process = Popen(full_command, stdout=temp_file,
                            universal_newlines=True)
        except OSError as exc:
            if exc.errno == errno.ENOENT:
                logging.error('Error: please make sure that rendercheck '
                              'is installed.')
                exit(1)
            else:
                raise

Problem is, if I read that correctly, assuming Jeffrey's case is correct and rendercheck is not present, it "should" throw ENOENT (doesn't exist) and log a message and exit with a 1.

So logging.error, IIRC logs to wherever the logger is suppose to log, file or stdout, logging.error isn't internally redirected to stderr.

Second, it should exit with a 1, which should trigger the fail message, not a pass based on this command in the job description:

command: rendercheck_test -d -o $CHECKBOX_DATA/rendercheck-results && echo "Rendercheck tests completed successfully" || echo "Rendercheck completed, but there are errors. Please see the log $CHECKBOX_DATA/rendercheck-results for details"

which gives me this when I move the rendercheck tool:

bladernr@klaatu:~/development/checkbox$ which rendercheck
/usr/bin/rendercheck
bladernr@klaatu:~/development/checkbox$ sudo mv /usr/bin/rendercheck /usr/bin/not-the-rendercheck-youre-looking-for
[sudo] password for bladernr:
bladernr@klaatu:~/development/checkbox$ which rendercheck
bladernr@klaatu:~/development/checkbox$ ./scripts/rendercheck_test -d -o /tmp/rendercheck.log
Traceback (most recent call last):
  File "./scripts/rendercheck_test", line 134, in get_suites_list
    stderr=PIPE, universal_newlines=True)
  File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.2/subprocess.py", line 1361, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'rendercheck'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./scripts/rendercheck_test", line 255, in <module>
    main()
  File "./scripts/rendercheck_test", line 165, in main
    all_tests = RenderCheck().get_suites_list()
  File "./scripts/rendercheck_test", line 136, in get_suites_list
    if exc.errno == errno.ENOENT:
NameError: global name 'errno' is not defined