"recent" page shows internal server error

My method:
 * Login as cbuild@toolchain64
 * cd lib/tcwg-web
 * edit tcwg-web.ini
 * set development=True
 * python index.py
 * wget http://toolchain64:8080/helpers/recent

See this stack trace:

cbuild@cbuild-master:~/lib/tcwg-web$ jed tcwg-web.ini
cbuild@cbuild-master:~/lib/tcwg-web$ python index.py
http://0.0.0.0:8080/
filename = /home/cbuild/public_html/build/builds.pickle
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/web/application.py", line 242, in process
    return self.handle()
  File "/usr/lib/pymodules/python2.7/web/application.py", line 233, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/pymodules/python2.7/web/application.py", line 415, in _delegate
    return handle_class(cls)
  File "/usr/lib/pymodules/python2.7/web/application.py", line 390, in handle_class
    return tocall(*args)
  File "/home/cbuild/lib/tcwg-web.lava/recent.py", line 112, in GET
    builds = load()
  File "/home/cbuild/lib/tcwg-web.lava/recent.py", line 102, in load
    ret = load1('build')
  File "/home/cbuild/lib/tcwg-web.lava/recent.py", line 67, in load1
    top = common.load_builds(context)
  File "/home/cbuild/lib/tcwg-web.lava/common.py", line 100, in load_builds
    builds = cPickle.load(f)
EOFError

So builds.pickle is invalid in some way.

You *must* remember to unset the development flag and kill python, else nginx will show a gateway timeout.

build@cbuild-master:~/lib/tcwg-web$ ls -la ~/public_html/build/bu*

-rw-r--r-- 1 cbuild lp-users 1302528 Mar 20 07:30 /home/cbuild/public_html/build/builds.json
-rw-r--r-- 1 cbuild lp-users 0 Mar 18 11:47 /home/cbuild/public_html/build/builds.json.gz
-rw-r--r-- 1 cbuild lp-users 0 Mar 21 18:46 /home/cbuild/public_html/build/builds.pickle

So builds.pickle is zero length for some reason.

Running

cbuild@cbuild-master:~/public_html/build$ python ~/lib/cbuild-tools/cache-builds.py .

Traceback (most recent call last):
  File "/home/cbuild/lib/cbuild-tools/cache-builds.py", line 131, in <module>
    main()
  File "/home/cbuild/lib/cbuild-tools/cache-builds.py", line 128, in main
    json.dump(top, f, indent=2)
  File "/usr/lib/python2.7/json/__init__.py", line 182, in dump
    fp.write(chunk)
IOError: [Errno 28] No space left on device

As the disk is full:

cbuild@cbuild-master:~/public_html/build$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 51613624 24436120 24557124 50% /
udev 4085216 8 4085208 1% /dev
tmpfs 1635704 232 1635472 1% /run
none 5120 0 5120 0% /run/lock
none 4089252 0 4089252 0% /run/shm
/dev/vdb 83886080 82567268 1276 100% /space

So the prime candidate for a clean up is ~/public_html/binaries. This has the final binary tarballs after each build and should be reaped now and again.

cbuild@cbuild-master:~/public_html/binaries$ find . -name "*~*" -mtime +30 -print -exec rm {} \;

deletes all user builds (~) more than 30 days old.