Comment 23 for bug 78538

The error ("Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox, or restart your system.") appears because FF can't lock the profile. Why that is I don't know.

http://lxr.mozilla.org/seamonkey/source/toolkit/xre/nsAppRunner.cpp#2861

SelectProfile(), tries to lock the profile and if that fails then it calls ProfileLockedDialog() which displays the error (both functions are defined in the same source file).
But I didn't find the code that kicks in when FF sees that an instance is already running and connects to it (I suppose such code exists so that FF can avoid trying to lock the profile twice).

How can I profile FF? Is there something like strace so I could see which functions are being called?