Heap exhausted, large arrays

It seems to be a combination compiler/gc bug, because I don't run out of memory when I take the declaration out of the definition of v*. So, consider the attached code. Just load it into sbcl to see the crash.

Take a look at the total bytes allocated in the gc log files with gnuplot, and a command like:

plot "< awk '/Total bytes/ {n++ ; print n, $5}' testgc4-ok.log" w l, "< awk '/Total bytes/ {n++ ; print n, $5}' testgc4-crash.log" w l

Without the declaration, it doesn't grow too much over time. With the declaration, it keeps ratcheting up until the heap is exhausted.

I just built SBCL 1.2.10.72-c2a4aec, and it's not as bad. It can survive many calls to testcrash with 1mb arrays, and appears to use less memory in general. However, it's still the case that far more memory is used when the declaration is used, and it crashes pretty quickly when using 2mb arrays.

Also, I find the amount of garbage being retained to be surprising. A 2mb array of doubles is just 16mb, and only 3 are active at any time, which accounts for 48mb, yet the memory usage is becoming as 700mb. Use 5mb arrays instead, for a footprint of 120mb, and it quickly is unable to GC & crashes.

I posted this already in comp.lang.lisp, but I'll re-post it here.

I have a program which demonstrates this issue.
Maybe I'm really allocating more memory than my machine has?

(defclass cell ()
  ((data :initarg :data)))

(defun test-allocate (x)
  (declare (optimize (safety 3) (debug 3) (space 0) (speed 0)))

  (let ((w 2562)
        (h 1602)
        (d 3))
    (let ((arr2d (make-array (list w h)))
          (arr3d (make-array (list w h d))))

      (dotimes (i w)
        (dotimes (j h)
          (let ((vec (make-array (list d))))
            (let ((value (make-instance 'cell :data vec)))
              (setf (aref arr2d i j) value)
              (dotimes (k d)
                (setf (aref vec k) x))))))

      (dotimes (i w)
        (dotimes (j h)
          (dotimes (k d)
            (setf (aref arr3d i j k) x))))

      (list arr2d arr3d))))

(test-allocate 3.1)

* Heap exhausted during garbage collection: 0 bytes available, 32 requested.
 Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB LUB !move Alloc Waste Trig WP GCs Mem-age
   0: 0 0 0 0 0 0 0 0 0 0 0 10737418 0 0 0.0000
   1: 0 0 0 0 0 0 0 0 0 0 0 10737418 0 0 0.0000
   2: 1515 0 0 0 14529 90 0 0 43 478264576 770816 2000000 0 0 2.1092
   3: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   4: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   5: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   6: 0 0 0 0 1220 241 0 0 0 47874048 0 2000000 1092 0 0.0000
   Total bytes allocated = 1072272272
   Dynamic-space-size bytes = 1073741824
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = true
fatal error encountered in SBCL pid 9337:

Thanks Pascal Bourguignon for helping me find what seems to be a workaround.
If I start sbcl with the command line as follows, the test seems to run without exhausting the heap.

csh> sbcl --dynamic-space-size 3741824

I'm not sure what the original heap size was. From the error message output perhaps it was
(Dynamic-space-size bytes) 1073741824. In which case 3741824 is two orders of magnitude smaller.

when i write code, for clarity i keep such values in their units, unless there's a good reason not to. (it's a tough call in this case whether the unit is bits or bytes though...)

but --dynamic-space-size uses megabytes.

http://www.sbcl.org/manual/#Runtime-Options

Here is another use case. Basically: allocate a TEMPORARY string of 200M characters, (it's not referenced after the form is evaluated, not even in *); then allocate a vector of 200M octets. There is no exhaustion, because the 200M character string is garbage and it should be collected SILENTLY!

[pjb@despina :0.0 ~]$ sbcl --no-userinit
This is SBCL 1.4.1, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* (let ((string (make-string (* 200 1024 1024) :initial-element #\a))
        (pattern "07123E1F482356C415F684407A3B8723E10B2CBBC0B8FCD6282C49D37C9C1ABC"))
     (replace string pattern :start1 (- (length string) (length pattern)))
     (time (search pattern string)))

Evaluation took:
  2.012 seconds of real time
  2.010401 seconds of total run time (2.007339 user, 0.003062 system)
  99.90% CPU
  8,065,801,078 processor cycles
  0 bytes consed

209715136
* (let ((string (make-array (* 200 1024 1024) :element-type '(unsigned-byte 8) :initial-element (char-code #\a)))
        (pattern (map '(vector (unsigned-byte 8)) 'char-code "07123E1F482356C415F684407A3B8723E10B2CBBC0B8FCD6282C49D37C9C1ABC")))
    (replace string pattern :start1 (- (length string) (length pattern)))
    (time (search pattern string)))
Heap exhausted during allocation: 204046336 bytes available, 209715216 requested.
 Gen StaPg UbSta LaSta Boxed Unbox LB LUB !move Alloc Waste Trig WP GCs Mem-age
   0: 26538 0 0 45 0 0 0 0 1405408 69152 10737418 0 0 0.0000
   1: 26524 26512 0 19 2 0 25601 25608 839364224 217472 2000000 7 0 0.0000
   2: 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   3: 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   4: 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   5: 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   6: 0 0 0 570 304 0 0 0 27930768 708464 2000000 553 0 0.0000
   7: 875 876 0 0 0 0 0 0 0 0 2000000 0 0 0.0000
   Total bytes allocated = 868700400
   Dynamic-space-size bytes = 1073741824
GC control variables:
   *GC-INHIBIT* = false
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false

debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread
#<THREAD "main thread" RUNNING {1001B50083}>:
  Heap exhausted (no more space for allocation).
204046336 bytes available, 209715216 requested.

PROCEED WITH CAUTION.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit debugger, returning to top level.

(SB-KERNEL::HEAP-EXHAUSTED-ERROR 102023168 104857608)
0] :a

* (let ((string (make-array (* 200 1024 1024) :element-type '(unsigned-byte 8...