1) Patch CentOS init scripts for API services to apply some additional checks/kill of stuck service in stop() function (the easiest and fastest way).
2) Reconnfigure all local logging on OS nodes to log via TCP/UDP intead of /dev/log (would require refactoring of our rsyslog config files since running through UDP breaks log matching).
3) Find and apply a patch for the problem python module so python logging could handle recreation of /dev/log fine.
I see 3 possible ways to fix this:
1) Patch CentOS init scripts for API services to apply some additional checks/kill of stuck service in stop() function (the easiest and fastest way).
2) Reconnfigure all local logging on OS nodes to log via TCP/UDP intead of /dev/log (would require refactoring of our rsyslog config files since running through UDP breaks log matching).
3) Find and apply a patch for the problem python module so python logging could handle recreation of /dev/log fine.