Add support for compressed cores
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Apport |
New
|
Undecided
|
Unassigned | ||
| Daisy |
Confirmed
|
Wishlist
|
Unassigned | ||
| Whoopsie |
New
|
Undecided
|
Unassigned | ||
Bug Description
Hi,
Currently, we have a pretty large backlog of cores to process, the cores directory is at 184G. Would it be possible to add support for compressing the cores?
Randomly picking out some of the larger cores and compressing (gzip with default compression level) them seems to suggest we'd save around 50%:
compressed uncompressed ratio uncompressed_name
732576993 3083538432 76.2% 1c94ac3e-
15570174 220979200 93.0% 239a4308-
827197319 1093217311 24.3% 2627d290-
837838177 1108439573 24.4% 41413548-
960665967 1282747535 25.1% 4a83d5d8-
240154514 317718528 24.4% 52f12f5e-
680541715 2829602816 75.9% 8da28ff6-
666078076 887166520 24.9% 8da28ff6-
271885158 365674049 25.6% 932a1678-
179878866 245787972 26.8% b44d72d2-
129184094 171881716 24.8% becba9b4-
5541571053 11606753652 52.3% (totals)
I'm not sure how cores are uploaded, but if we could have the clients compress them, we'd also save on network traffic and more importantly, disk I/O.
| tags: | added: canonical-webops-eng |
| Changed in daisy: | |
| importance: | Undecided → Wishlist |
Hi Haw,
The data we receive from the clients is gzip'ed and base64'ed (because apport wants to produce text-only report files).
This seems wasteful, as your findings suggest. We should be decoding the base64 data client-side and just sending the gzip data, or possibly using a different compression algorithm in apport (and whoopsie). I briefly looked at lz4, and its higher compression ratio option looks promising.
The *.core files are the uncompressed core files and are either leftover from crashed retracers, or retraces-