Drizzle client crash if drizzled is restarted and client had default db

This is what valgrind shows at the time of the crash

==15267== Conditional jump or move depends on uninitialised value(s)
==15267== at 0x4C228E3: drizzle_result_free (result.c:80)
==15267== by 0x408551: build_completion_hash(bool, bool) (drizzle.cc:2471)
==15267== by 0x40B423: com_rehash(std::string*, char const*) (drizzle.cc:3574)
==15267== by 0x408B11: reconnect() (drizzle.cc:2582)
==15267== by 0x408CCF: drizzleclient_real_query_for_lazy(char const*, int, drizzle_result_st*, unsigned int*) (drizzle.cc:2643)
==15267== by 0x409139: com_go(std::string*, char const*) (drizzle.cc:2742)
==15267== by 0x4078E6: add_line(std::string*, char*, char*, bool*) (drizzle.cc:2157)
==15267== by 0x406D29: read_and_execute(bool) (drizzle.cc:1915)
==15267== by 0x405D4D: main (drizzle.cc:1324)
==15267==
==15267== Use of uninitialised value of size 8
==15267== at 0x4C23A10: drizzle_column_free (column.c:357)
==15267== by 0x4C228EC: drizzle_result_free (result.c:81)
==15267== by 0x408551: build_completion_hash(bool, bool) (drizzle.cc:2471)
==15267== by 0x40B423: com_rehash(std::string*, char const*) (drizzle.cc:3574)
==15267== by 0x408B11: reconnect() (drizzle.cc:2582)
==15267== by 0x408CCF: drizzleclient_real_query_for_lazy(char const*, int, drizzle_result_st*, unsigned int*) (drizzle.cc:2643)
==15267== by 0x409139: com_go(std::string*, char const*) (drizzle.cc:2742)
==15267== by 0x4078E6: add_line(std::string*, char*, char*, bool*) (drizzle.cc:2157)
==15267== by 0x406D29: read_and_execute(bool) (drizzle.cc:1915)
==15267== by 0x405D4D: main (drizzle.cc:1324)
==15267==
==15267== Invalid read of size 8
==15267== at 0x4C23A10: drizzle_column_free (column.c:357)
==15267== by 0x4C228EC: drizzle_result_free (result.c:81)
==15267== by 0x408551: build_completion_hash(bool, bool) (drizzle.cc:2471)
==15267== by 0x40B423: com_rehash(std::string*, char const*) (drizzle.cc:3574)
==15267== by 0x408B11: reconnect() (drizzle.cc:2582)
==15267== by 0x408CCF: drizzleclient_real_query_for_lazy(char const*, int, drizzle_result_st*, unsigned int*) (drizzle.cc:2643)
==15267== by 0x409139: com_go(std::string*, char const*) (drizzle.cc:2742)
==15267== by 0x4078E6: add_line(std::string*, char*, char*, bool*) (drizzle.cc:2157)
==15267== by 0x406D29: read_and_execute(bool) (drizzle.cc:1915)
==15267== by 0x405D4D: main (drizzle.cc:1324)
==15267== Address 0x15b182e160 is not stack'd, malloc'd or (recently) free'd
==15267==
==15267== Process terminating with default action of signal 11 (SIGSEGV)
==15267== Access not within mapped region at address 0x15B182E160
==15267== at 0x4C23A10: drizzle_column_free (column.c:357)
==15267== by 0x4C228EC: drizzle_result_free (result.c:81)
==15267== by 0x408551: build_completion_hash(bool, bool) (drizzle.cc:2471)
==15267== by 0x40B423: com_rehash(std::string*, char const*) (drizzle.cc:3574)
==15267== by 0x408B11: reconnect() (drizzle.cc:2582)
==15267== by 0x408CCF: drizzleclient_real_query_for_lazy(char const*, int, drizzle_result_st*, unsigned int*) (drizzle.cc:2643)
==15267== by 0x409139: com_go(std::string*, char const*) (drizzle.cc:2742)
==15...

Hi!

So I had a quick look at this with a debugger at the lines of code that look to be the issue. Basically, the segfault occurs on this call:

drizzle_result_free(&databases);

The only way this method gets called is if drizzle_query_str() returns something which is not NULL. However, if we look at what the drizzle_query_str() method returns, it returns a pointer to a drizzle_result_st structure. This return value will never be NULL in this case because the actual object is passed as a parameter to the drizzle_query_str() function (the &ret parameter to this function). Thus, the if condition will always evaluate to true.

I was curious to know what the return value is after calling drizzle_query_str() and it turns out to be DRIZZLE_RETURN_COULD_NOT_CONNECT.

Now, when does that value get returned by libdrizzle? Well, it only gets returned in one place - the function drizzle_state_connect() has the following code block:

if (con->addrinfo_next == NULL)
{
  DRIZZLE_ERROR_SET(con->drizzle, "drizzle_state_connect",
                    "could not connect")
  DRIZZLE_STATE_RESET(con)
  return DRIZZLE_RETURN_COULD_NOT_CONNECT;
}

Thus, we can see that we are calling drizzle_query_str() which in turn calls drizzle_command_write() which then calls drizzle_con_connect(). This function pushes the drizzle_state_connect() function onto its state stack (which will eventually be popped off and called).

So I'm thinking a possible fix might be the following at (or around) line 2471 in client/drizzle.cc:

if (ret != DRIZZLE_RETURN_COULD_NOT_CONNECT)
{
  drizzle_result_free(&databases);
}
else
{
  return;
}

I've tried this out and it works but you might want to think a bit more on what I'm suggesting. I just had a quick look so maybe my analysis is incorrect and there is a better solution to the issue.

Another possible fix might be to change the offending block of code in client/drizzle.cc to:

/* hash all database names */
drizzle_query_str(&con, &databases, "show databases", &ret);
if (ret == DRIZZLE_RETURN_OK)
{
  if (drizzle_result_buffer(&databases) != DRIZZLE_RETURN_OK)
    put_info(drizzle_error(&drizzle),INFO_INFO,0,0);
  else
  {
    while ((database_row=drizzle_row_next(&databases)))
    {
      tmp_str= database_row[0];
      tmp_str_lower= lower_string(tmp_str);
      completion_map[tmp_str_lower]= tmp_str;
    }
  }
  drizzle_result_free(&databases);
}
else
{
  return;
}

I hope this information is useful to you.

-Padraig

Hi Padraig!

Thanks for looking into this.

This is something else I have found so far:

1- I start drizzled and I then start the client under gdb with a breakpoint on
client/drizzle.cc

static void build_completion_hash(bool rehash, bool write_info)

And I looked at "databases", I then restart drizzled and send
"use test;"
when I print databases, note the line with:

field = 0x6d7b1 <Address 0x6d7b1 out of bounds>

I was wondering if there was a way to "reset" databases to a "clean" state once we are here.

Thanks

-Diego

(gdb) run
Starting program: /Applications/mysql/enterprise/drizzle-local/bin/drizzle -p3306 -h192.168.0.99
warning: posix_spawn failed, trying execvp, error: 86

Breakpoint 1, build_completion_hash (rehash=true, write_info=true) at client/drizzle.cc:2429
2429 COMMANDS *cmd=commands;
(gdb) print databases
$4 = {
  con = 0x7fff5fbfe820,
  next = 0x1000a5bf0,
  prev = 0x5fbfe820,
  options = 446912,
  info = "\001", '\0' <repeats 11 times>, "@####\000\000\000\000\000\000\000\000\000\000@####\000\000\020####\000\000}3\n\000\001", '\0' <repeats 11 times>, "##\006\000\001\000\000\000##\006\000\001", '\0' <repeats 1974 times>,
  error_code = 0,
  sqlstate = "\000\000\000\000\000",
  insert_id = 0,
  warning_count = 0,
  affected_rows = 0,
  column_count = 0,
  column_current = 0,
  column_list = 0x0,
  column = 0x0,
  column_buffer = 0x0,
  row_count = 0,
  row_current = 0,
  field_current = 0,
  field_total = 0,
  field_offset = 0,
  field_size = 0,
  field = 0x0,
  field_buffer = 0x0,
  row_list_size = 0,
  row = 0x0,
  row_list = 0x0,
  field_sizes = 0x0,
  field_sizes_list = 0x0
}
(gdb) c
Continuing.
Welcome to the Drizzle client.. Commands end with ; or \g.
Your Drizzle connection id is 2
Server version: 2009.09.1127 Source distribution (local-trunk)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

<<<< Right here I restarted drizzled >>>>>

drizzle> use test;

Breakpoint 1, build_completion_hash (rehash=true, write_info=true) at client/drizzle.cc:2429
2429 COMMANDS *cmd=commands;
(gdb) print databases
$5 = {
  con = 0x0,
  next = 0x0,
  prev = 0x0,
  options = DRIZZLE_RESULT_NONE,
  info = '\0' <repeats 2047 times>,
  error_code = 0,
  sqlstate = "\000\000\000\000\000",
  insert_id = 4295414208,
  warning_count = 58416,
  affected_rows = 4295621717,
  column_count = 4,
  column_current = 0,
  column_list = 0x0,
  column = 0x7fff5fbfe45c,
  column_buffer = 0x10006d1c0,
  row_count = 0,
  row_current = 72057598332895233,
  field_current = 58480,
  field_total = 4295646192,
  field_offset = 0,
  field_size = 4295414208,
  field = 0x6d7b1 <Address 0x6d7b1 out of bounds>,
  field_buffer = 0x0,
  row_list_size = 4297457664,
  row = 0x1,
  row_list = 0x5,
  field_sizes = 0x0,
  field_sizes_list = 0x7fff5fbfe4c0
}
(gdb)

Just another thought I had last night, if there is no connection to drizzled, why call build_completion_hash()?

Well, the call to build_completion_hash() is optional. Whether it is called or not is determined by the startup options that are passed to drizzled.

The startup option that controls it is opt_rehash as you can see from this block:

if (opt_rehash)
  com_rehash(NULL, NULL);

Looking at the description of this option, we see:

{"auto-rehash", OPT_AUTO_REHASH,
   N_("Enable automatic rehashing. One doesn't need to use 'rehash' to get table and field completion, but startup and reconnecting may take a longer time. Disable with --disable-auto-rehash."),
   (char**) &opt_rehash, (char**) &opt_rehash, 0, GET_BOOL, NO_ARG, 1, 0, 0, 0,
   0, 0}

So if you pass the --disable-auto-rehash option to drizzled on startup, build_completion_hash() will not get called from the reconnect() method.

-Padraig

This is what I did and drizzle does not crash now, but there is a new crash :P

in client/drizzle.cc

static int reconnect(void)
{
  /* purecov: begin tested */
  if (opt_reconnect)
  {
    put_info(_("No connection. Trying to reconnect..."),INFO_INFO,0,0);
    (void) com_connect((string *)0, 0);
- if (opt_rehash)
+ if (opt_rehash && connected)
      com_rehash(NULL, NULL);
  }
  if (!connected)
    return put_info(_("Can't connect to the server\n"),INFO_ERROR,0,0);
  /* purecov: end */
  return 0;
}

I call com_rehash() only if we are connected, on a reconnect that fails, there should be no need to call com_rehash() (I think)

I'll update on the new crash later tonight.

Changed it to drizzle because the fix was done there.

The new crash happens following these steps: (of course. after applying the patch on this bug report )

1- Start drizzled
2- Connect to it using the command line client
3- Run this query:
drizzle> Use information_schema;
4- Stop drizzled
5- Run:
drizzle> Use information_schema;

Boom! segfault.