chef server returned 500 internal server error when uploading cookbooks

Bug #1397309 reported by Jerry Zhao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Compass
In Progress
Undecided
Unassigned

Bug Description

2014-11-28 03:34:19.989 | Uploading openstack-object-storage [9.0.3]
2014-11-28 03:34:19.994 | Uploading openstack-ops-database [9.0.1]
2014-11-28 03:34:19.996 | Uploading openstack-ops-messaging [9.0.1]
2014-11-28 03:34:19.997 | Uploading openstack-orchestration [9.1.5]
2014-11-28 03:34:20.000 | Uploading openstack-telemetry [9.1.1]
2014-11-28 03:34:20.002 | Uploading pacman [1.1.1]
2014-11-28 03:34:20.004 | Uploading postgresql [3.3.4]
2014-11-28 03:34:20.024 | Uploading python [1.4.6]
2014-11-28 03:34:20.032 | Uploading rabbitmq [3.0.4]
2014-11-28 03:34:20.034 | Uploading rsyslog [1.0.0]
2014-11-28 03:34:20.036 | Uploading runit [1.5.10]
2014-11-28 03:34:20.038 | Uploading selinux [0.7.2]
2014-11-28 03:34:20.040 | Uploading statsd [0.1.6]
2014-11-28 03:34:20.041 | Uploading sysctl [0.1.15]
2014-11-28 03:34:20.042 | Uploading windows [1.31.0]
2014-11-28 03:34:20.047 | Uploading xfs [1.1.0]
2014-11-28 03:34:20.047 | Uploading yum [3.1.4]
2014-11-28 03:34:20.049 | Uploading yum-epel [0.3.4]
2014-11-28 03:34:20.051 | Uploading yum-erlang_solutions [0.2.0]
2014-11-28 03:34:41.566 | ERROR: internal server error
2014-11-28 03:34:41.566 | Response: internal service error

The issue only happened to tests on vm sometimes. Don't know if a upgrade chef server would fix the problem.

Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :

https://tickets.opscode.com/browse/CHEF-5277
someone else is also seeing this.

Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :

another helpful bug report:
https://tickets.opscode.com/browse/CHEF-4504

it seems to be fixed in chef server 11.1.
I will write a patch setting the concurrency to 1 to work around it before we upgrade chef-server to R11 and up.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to compass-core (dev/experimental)

Reviewed: https://review.openstack.org/137731
Committed: https://git.openstack.org/cgit/stackforge/compass-core/commit/?id=a5259401e229b7153be35c14b19a9b3f16e06fa0
Submitter: Jenkins
Branch: dev/experimental

commit a5259401e229b7153be35c14b19a9b3f16e06fa0
Author: zhaoxinyu <email address hidden>
Date: Thu Nov 27 23:27:25 2014 -0800

    add status check and timeout for fastesturl

    also simplified some code to retrieve a result.
    add huawei source for chef-server and client packages.
    upgrade chef-client to 11.10 to take advantage of
    concurrency option in knife cookbook upload.
    Change-Id: Ic2c771b377918622da88f7243374056d5e0b3dc2
    Partial-bug: #1397309

syang1 (shuo-yang)
Changed in compass:
status: New → Fix Committed
Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :

++ '[' 0 -ne 0 ']'
++ knife cookbook upload --all --cookbook-path /var/chef/cookbooks --concurrency 1
Uploading apache2 [1.9.6]
Uploading apt [2.3.8]
Uploading aws [2.1.1]
Uploading build-essential [1.4.2]
Uploading ceph [0.2.1]
Uploading chef_handler [1.1.6]
Uploading collectd [1.0.3]
Uploading cpu [0.2.0]
Uploading database [2.0.0]
Uploading dmg [2.2.0]
Uploading erlang [1.4.2]
Uploading git [4.0.2]
Uploading haproxy [1.6.2]
Uploading homebrew [1.6.6]
Uploading iptables [0.13.2]
Uploading keepalived [1.2.0]
Uploading logrotate [1.5.0]
Uploading lvm [1.0.5]
Uploading memcached [1.7.2]
Uploading mysql [4.1.2]
Uploading nagios [0.0.0]
Uploading net [1.0.0]
Uploading nodejs [1.3.0]
Uploading ntp [1.4.1]
Uploading ohai [1.1.13]
Uploading openssh [1.3.5]
Uploading openssl [1.1.0]
Uploading openstack-block-storage [9.2.3]
Uploading openstack-common [9.4.1]
Uploading openstack-compute [9.2.2]
Uploading openstack-dashboard [9.0.3]
Uploading openstack-identity [9.2.1]
Uploading openstack-image [9.1.1]
Uploading openstack-network [9.0.8]
Uploading openstack-object-storage [9.0.3]
Uploading openstack-ops-database [9.0.1]
Uploading openstack-ops-messaging [9.0.1]
Uploading openstack-orchestration [9.1.5]
Uploading openstack-telemetry [9.1.1]
Uploading pacman [1.1.1]
Uploading postgresql [3.3.4]
Uploading python [1.4.6]
Uploading rabbitmq [3.0.4]
Uploading rsyslog [1.0.0]
Uploading runit [1.5.10]
Uploading selinux [0.7.2]
Uploading statsd [0.1.6]
Uploading sysctl [0.1.15]
Uploading windows [1.31.0]
Uploading xfs [1.1.0]
Uploading yum [3.1.4]
Uploading yum-epel [0.3.4]
Uploading yum-erlang_solutions [0.2.0]
ERROR: internal server error
Response: internal service error
++ [[ 100 != \0 ]]
++ echo 'failed to add cookbooks

The bug appeared again. Upgrading Chef-server may be our last straw

Changed in compass:
status: Fix Committed → In Progress
Revision history for this message
Jerry Zhao (zhaoxinyu) wrote :

reopen the bug

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.