Juniper Openstack

Openstack HA , sshd directory getting removed from /var/run and mysql syncroniztion is not happening.

Bug #1398581 reported by venu kolli
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Released
High
Prabhakaran Ganesan
R2.0
Fix Released
High
Prabhakaran Ganesan
Juniper Openstack r2.01
R2.1
Fix Released
Critical
Sanju Abraham

Bug Description

Observed an instance where sshd directory is removed from /var/run/ from two of the control nodes and both of them are not accessible.

What i find from auth logs is sshd directory got removed , please find the logs from auth.log

Nov 6 01:26:18 b4s342 su[19125]: Successful su for rabbitmq by root
Nov 6 01:26:18 b4s342 su[19123]: + ??? root:rabbitmq
Nov 6 01:26:18 b4s342 su[19125]: + ??? root:rabbitmq
Nov 6 01:26:18 b4s342 su[19125]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:26:18 b4s342 su[19123]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:26:18 b4s342 su[19123]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:26:18 b4s342 su[19125]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:27:23 b4s342 su[30638]: Successful su for rabbitmq by root
Nov 6 01:27:23 b4s342 su[30638]: + ??? root:rabbitmq
Nov 6 01:27:23 b4s342 su[30639]: Successful su for rabbitmq by root
Nov 6 01:27:23 b4s342 su[30639]: + ??? root:rabbitmq
Nov 6 01:27:23 b4s342 su[30638]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:27:23 b4s342 su[30639]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:27:23 b4s342 su[30639]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:27:23 b4s342 su[30638]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:28:29 b4s342 su[9165]: Successful su for rabbitmq by root
Nov 6 01:28:29 b4s342 su[9165]: + ??? root:rabbitmq
Nov 6 01:28:29 b4s342 su[9166]: Successful su for rabbitmq by root
Nov 6 01:28:29 b4s342 su[9166]: + ??? root:rabbitmq
Nov 6 01:28:29 b4s342 su[9165]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:28:29 b4s342 su[9166]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:28:29 b4s342 su[9165]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:28:29 b4s342 su[9166]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:29:34 b4s342 su[21258]: Successful su for rabbitmq by root
Nov 6 01:29:34 b4s342 su[21258]: + ??? root:rabbitmq
Nov 6 01:29:34 b4s342 su[21258]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:29:34 b4s342 su[21260]: Successful su for rabbitmq by root
Nov 6 01:29:34 b4s342 su[21260]: + ??? root:rabbitmq
Nov 6 01:29:34 b4s342 su[21260]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Nov 6 01:29:34 b4s342 su[21260]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:29:34 b4s342 su[21258]: pam_unix(su:session): session closed for user rabbitmq
Nov 6 01:30:27 b4s342 sshd[30878]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:27 b4s342 sshd[30879]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:27 b4s342 sshd[30880]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:27 b4s342 sshd[30881]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30889]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30890]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30891]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30892]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30894]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30895]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30898]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:28 b4s342 sshd[30899]: fatal: Missing privilege separation directory: /var/run/sshd
Nov 6 01:30:40 b4s342 su[811]: Successful su for rabbitmq by root
Nov 6 01:30:40 b4s342 su[811]: + ??? root:rabbitmq
Nov 6 01:30:40 b4s342 su[812]: Successful su for rabbitmq by root
Nov 6 01:30:40 b4s342 su[812]: + ??? root:rabbitmq

See original description
Tags: ha
venu kolli (vkolli)
Changed in juniperopenstack:
assignee: nobody → Prabhakaran Ganesan (gprabhak)
importance: Undecided → High
milestone: none → r1.30-fcs
venu kolli (vkolli)
Changed in juniperopenstack:
milestone: r1.30-fcs → none
Revision history for this message
venu kolli (vkolli) wrote :

Going to take care of this issue is on mainline

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/7372
Committed: http://github.org/Juniper/contrail-provisioning/commit/0d3759f3a170e7fee4619a95e7b822cdb71d0c6f
Submitter: Zuul
Branch: master

commit 0d3759f3a170e7fee4619a95e7b822cdb71d0c6f
Author: Ganesan Prabhakaran <email address hidden>
Date: Thu Feb 12 11:00:55 2015 -0800

/var/run/sshd gets deleted resulting it SSH being disallowed to the hosts
This also results in galera cluster sync issues. Adding this temporary
workaround to create the directory from hamon backgroud task

Change-Id: I7193cfb32b71fe22f6de9abc90c5647674ca6392
Closes-bug:1398581

Changed in juniperopenstack:
status: New → Fix Committed
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7373
Committed: http://github.org/Juniper/contrail-provisioning/commit/5313dbf913049b7deac7dd11a93950a85e135796
Submitter: Zuul
Branch: R2.0

commit 5313dbf913049b7deac7dd11a93950a85e135796
Author: Ganesan Prabhakaran <email address hidden>
Date: Thu Feb 12 11:19:27 2015 -0800

/var/run/sshd gets deleted resulting it SSH being disallowed to the hosts
This also results in galera cluster sync issues. Adding this temporary
workaround to create the directory from hamon backgroud task

Change-Id: I2074068866c5e7505c06c073b9eafc99ce1bce35
Closes-bug: 1398581

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7382
Committed: http://github.org/Juniper/contrail-provisioning/commit/a9c8f39a3157397404d1c8255fc970d883718235
Submitter: Zuul
Branch: R2.1

commit a9c8f39a3157397404d1c8255fc970d883718235
Author: Ganesan Prabhakaran <email address hidden>
Date: Thu Feb 12 14:13:56 2015 -0800

/var/run/sshd gets deleted resulting it SSH being disallowed to the hosts
This also results in galera cluster sync issues. Adding this temporary
workaround to create the directory from hamon backgroud task

Change-Id: I13630bb2c3711b74dfc256eedaea2fb581f5a755
Closes-bug: 1398581

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7926
Committed: http://github.org/Juniper/contrail-provisioning/commit/6d2b8b73c0cbd84bb6bbc2efb5a4d92d461c17a9
Submitter: Zuul
Branch: R2.0

commit 6d2b8b73c0cbd84bb6bbc2efb5a4d92d461c17a9
Author: Sanju Abraham <email address hidden>
Date: Fri Feb 27 20:35:54 2015 -0800

Close-Bug:1398581. This bug fix addresses the issue of ssh run dir that was being deleted by cmon since cmon's run dir was /var/run. A purge job that is configured not by cron but bya process internal would clean up all empty dirs that are older than 7 days. The fix will ensure the run dir for cmon is /var/run/cmon

Change-Id: I633b1f937f9d0e654d0f3592da715c5d439ae56a

Revision history for this message
Sanju Abraham (asanju) wrote :

This bug fix addresses the issue of ssh run dir that was being deleted by cmon since cmon's run dir was /var/run. A purge job that is configured not by cron but bya process internal would clean up all empty dirs that are older than 7 days. The fix will ensure the run dir for cmon is /var/run/cmon

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7925
Committed: http://github.org/Juniper/contrail-provisioning/commit/d82d447debd5fb7b0780187c688c11f1c5cb2408
Submitter: Zuul
Branch: R2.1

commit d82d447debd5fb7b0780187c688c11f1c5cb2408
Author: Sanju Abraham <email address hidden>
Date: Fri Feb 27 20:19:17 2015 -0800

Close-Bug:1398581. This bug fix addresses the issue of ssh run dir that was being deleted by cmon since cmon's run dir was /var/run. A purge job that is configured not by cron but bya process internal would clean up all empty dirs that are older than 7 days. The fix will ensure the run dir for cmon is /var/run/cmon

Change-Id: I4b08f25fc1192838190415d12e522b5029063200

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7929
Committed: http://github.org/Juniper/contrail-provisioning/commit/5c7b14843e063fa5819517b52e69255a97d33642
Submitter: Zuul
Branch: R2.1

commit 5c7b14843e063fa5819517b52e69255a97d33642
Author: Sanju Abraham <email address hidden>
Date: Sat Feb 28 12:48:44 2015 -0800

Closes-Bug:1398581. The workaround to create /var/run/sshd was provided to circumvent an issue that was not known earlier. The actual fix is provided in https://review.opencontrail.org/#/c/7926 and hence removing the periodic check for sshd and re-creating it

Change-Id: I5ef9171449d7fd18e7514e46d9e401b48f75b407

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/7930
Committed: http://github.org/Juniper/contrail-provisioning/commit/f37364588be5e12e4e55199f3d658e6bfe42c9c0
Submitter: Zuul
Branch: R2.0

commit f37364588be5e12e4e55199f3d658e6bfe42c9c0
Author: Sanju Abraham <email address hidden>
Date: Sat Feb 28 12:52:04 2015 -0800

Closes-Bug:#1398581. The workaround to create /var/run/sshd was provided to circumvent an issue that was not known earlier. The actual fix is provided in https://review.opencontrail.org/#/c/7926 and hence removing the periodic check for sshd and re-creating it

Change-Id: Iead22ac4deb69b0e9fba5f910c74d1c5b94df454

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : master

Review in progress for https://review.opencontrail.org/8080

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/8080
Committed: http://github.org/Juniper/contrail-provisioning/commit/9d5f728ef8f091116233a5921e1508db0b03e5ed
Submitter: Zuul
Branch: master

commit 9d5f728ef8f091116233a5921e1508db0b03e5ed
Author: Sanju Abraham <email address hidden>
Date: Wed Mar 4 17:56:59 2015 -0800

Closes-Bug:#1398581. This bug fix addresses the issue of ssh run dir that was being deleted by cmon since cmon's run dir was /var/run.

Change-Id: Ia36c08347f9adb0c2f7cae7d98002dfd33041dc1

Prabhakaran Ganesan (gprabhak)
description: updated
information type: Proprietary → Public
venu kolli (vkolli)
Changed in juniperopenstack:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.