OpenStack Core Infrastructure

Grenade fails with /opt/stack/logs/screen/error.log: Permission denied

Bug #1355899 reported by Dan Smith
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceilometer
Invalid
Undecided
Unassigned
Ceilometer next
OpenStack Core Infrastructure
Incomplete
Undecided
Unassigned
grenade
Invalid
Undecided
Unassigned

Bug Description

Grenade fails to take action on some openstack-owned log file due to a permissions error:

http://logs.openstack.org/37/113337/2/check/check-grenade-dsvm-partial-ncpu/d375363/logs/grenade.sh.txt.gz

2014-08-12 14:27:01.606 | + screen -S stack -X screen -t ceilometer-anotification
2014-08-12 14:27:01.610 | There is a screen on:
2014-08-12 14:27:01.610 | 29046.stack (08/12/2014 02:07:45 PM) (Dead ???)
2014-08-12 14:27:01.611 | Remove dead screens with 'screen -wipe'.
2014-08-12 14:27:01.611 | No screen session found.
2014-08-12 14:27:01.612 | + die 91 'Failure in upgrade-ceilometer'
2014-08-12 14:27:01.612 | + local exitcode=1
2014-08-12 14:27:01.612 | [Call Trace]
2014-08-12 14:27:01.612 | ./grenade.sh:323:upgrade_service
2014-08-12 14:27:01.612 | ./grenade.sh:91:die
2014-08-12 14:27:01.615 | [ERROR] ./grenade.sh:91 Failure in upgrade-ceilometer
2014-08-12 14:27:01.615 | /opt/stack/new/grenade/functions: line 133: /opt/stack/logs/screen/error.log: Permission denied
2014-08-12 14:27:01.615 | Exit code: 1
2014-08-12 14:27:01.643 | World dumping... see /opt/stack/logs/worlddump-2014-08-12-142701.txt for details

Revision history for this message
Clark Boylan (cboylan) wrote :

This appears to have failed because ceilometer failed to upgrade. Need more info from grenade and ceilometer if this is an actual infra bug.

Changed in openstack-ci:
status: New → Incomplete
Revision history for this message
gordon chung (chungg) wrote :

seems pretty intermittent:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIkZhaWx1cmUgaW4gdXBncmFkZS1jZWlsb21ldGVyXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDkzNDcxMzg2ODF9

Revision history for this message
gordon chung (chungg) wrote :

this hasn't occurred recently (assuming my logstash query is correct)... i'll leave this open for now so we have a placeholder if it does come back.

Eoghan Glynn (eglynn)
Changed in ceilometer:
status: New → Triaged
assignee: nobody → Chris Dent (chdent)
milestone: none → next
Revision history for this message
Chris Dent (cdent) wrote :

It's not always happening on ceilometer. It's basically anything that can't start a screen because a screen was left behind, try this query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiUmVtb3ZlIGRlYWQgc2NyZWVucyB3aXRoICdzY3JlZW4gLXdpcGUnXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MTE5ODkxMTcyOTZ9

Presumably once the USE_SCREEN=False stuff is turned on we'll be able to inspect this more closely.

In any case, this doesn't appear to be a ceilo problem.

Revision history for this message
Chris Dent (cdent) wrote :

A recent look at logstash[1] shows that this is still happening every now and again, even with the USE_SCREEN changes in place. Digging around it appears the problem here is that the value of SCREEN_LOGDIR is different when the err() function is called from when it is used to create the the SCREEN_LOGDIRs.

In a grenade.sh log we see these various assignments:

SCREEN_LOGDIR=/opt/stack/new/logs/screen
SCREEN_LOGDIR=/opt/stack/new/screen-logs
SCREEN_LOGDIR=/opt/stack/old/logs/screen
SCREEN_LOGDIR=/opt/stack/old/screen-logs

But the one that is actually used when calling err() is '/opt/stack/logs/screen/error.log' which is set in grenaderc.

Given the intricacies of where in the stack we are and who is in charge of a env variable at any given moment I'm not sure what the right fix is here.

Note: This error is hiding actual errors, but is not the cause of failure a grenade run itself.

[1] http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiL29wdC9zdGFjay9sb2dzL3NjcmVlbi9lcnJvci5sb2c6IFBlcm1pc3Npb24gZGVuaWVkXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MTI1ODk0ODM5NzB9

Changed in ceilometer:
status: Triaged → Invalid
Revision history for this message
Chris Dent (cdent) wrote :

Actually this ought to fix it: https://review.openstack.org/#/c/125924/

However there may be another bug: Does /opt/stack/logs/screen/error.log need to be one of the stashed files?

Chris Dent (cdent)
Changed in ceilometer:
assignee: Chris Dent (cdent) → nobody
Revision history for this message
Sean Dague (sdague) wrote :

This grenade bug was last updated over 180 days ago, as grenade
is a fast moving project and we'd like to get the tracker down to
currently actionable bugs, this is getting marked as Invalid. If the
issue still exists, please feel free to reopen it.

Changed in grenade:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.