This issue was seen when we issued the provisioning multiple times ( more than 10 times) without making any change in the json file. The cmd below was used to provision the system:
/opt/contrail/contrail_server_manager/provision_containers.sh -ni -j /root/regress_provision_with_compute.json
The script setup_vnc_galera failed to restart MySQL on one of the servers and we see the following crash in MySQL logs. Even trying to manually start MySQL service "service mysql start" produces the same log.
When this command is executed on vm2 10.204.217.32, to bring up vnc_galera :-
MYSQL_ROOT_PW=B81CE7189419 setup-vnc-galera --self_ip 10.204.217.32 --keystone_ip 10.204.217.89 --galera_ip_list 10.204.217.31 10.204.217.32 10.204.217.33 --internal_vip 10.204.217.89 --zoo_ip_list 10.204.217.31 10.204.217.32 10.204.217.33 --keystone_user keystone --keystone_pass keystone --cmon_user cmon --cmon_pass cmon --monitor_galera True --openstack_index 2 --install_mysql_db False
The script setup_vnc_galera fails to restart MySQL and we see the following crash in MySQL logs. Even trying to manually start MySQL service "service mysql start" produces the same log.
so provisioning remained stuck at this stage
/opt/contrail/contrail_server_manager/provision_status.sh
+-------------+-------------------------------+---------------+-------------------+
| id | status | ip_address | mac_address |
+-------------+-------------------------------+---------------+-------------------+
| nodel11-vm1 | pre_exec_vnc_galera_completed | 10.204.217.31 | 52:53:58:01:00:01 |
| nodel11-vm2 | pre_exec_vnc_galera_started | 10.204.217.32 | 52:53:58:01:00:02 |
| nodel11-vm3 | openstack_completed | 10.204.217.33 | 52:53:58:01:00:03 |
| nodel11-vm4 | provision_issued | 10.204.217.34 | 52:53:58:01:00:04 |
| nodel11-vm5 | provision_issued | 10.204.217.36 | 52:53:58:01:00:05 |
+-------------+-------------------------------+---------------+-------------------+
171110 06:20:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
171110 06:20:15 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.aCPChCgITX
171110 06:20:18 mysqld_safe WSREP: Failed to recover position: 171110 6:20:15 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 171110 6:20:15 [Note] Plugin 'FEDERATED' is disabled. 171110 6:20:15 InnoDB: The InnoDB memory heap is disabled 171110 6:20:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins 171110 6:20:15 InnoDB: Compressed tables use zlib 1.2.3.3 171110 6:20:15 InnoDB: Using Linux native AIO 171110 6:20:15 InnoDB: Initializing buffer pool, size = 1.0G 171110 6:20:15 InnoDB: Completed initialization of buffer pool 171110 6:20:15 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 492773526 171110 6:20:15 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: ############### CORRUPT LOG RECORD FOUND InnoDB: Log record type 100, space id 56, page number 57 InnoDB: Log parsing proceeded successfully up to 492773526 InnoDB: Previous log record type 999999, is multi 0 InnoDB: Recv offset 0, prev 0 InnoDB: Hex dump of corrupt log starting 100 bytes before the start InnoDB: of the previous log rec, InnoDB: and ending 100 bytes after the start of the corrupt rec: len 200; hex 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000021020000000000064383965613665326234353563616266326133396339386335343939651f0800818f003800002898e02600818f00020002000080002a643c6664383965613665326234353563616266326133396339386335343939651f0200829400380501008294006a; asc d89ea6e2b455cabf2a39c98c5499e 8 ( & *d<fd89ea6e2b455cabf2a39c98c5499e 8 j; InnoDB: Set innodb_force_recovery to ignore this error. 171110 6:20:18 InnoDB: Assertion failure in thread 140362781017984 in file log0recv.c line 2200 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 06:20:18 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=25165824 read_buffer_size=131072 max_used_connections=0 max_threads=10000 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21904341 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x82a365] /usr/sbin/mysqld(handle_fatal_signal+0x403)[0x6af4b3] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fa8c171c330] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fa8bfd28c37] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fa8bfd2c028] /usr/sbin/mysqld[0x91ccbe] /usr/sbin/mysqld[0x92021a] /usr/sbin/mysqld[0x920af0] /usr/sbin/mysqld[0x879259] /usr/sbin/mysqld[0x84306d] /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x6b2548] /usr/sbin/mysqld[0x5ad75a] /usr/sbin/mysqld(_Z11plugin_initPiPPci+0xb5d)[0x5b0dfd] /usr/sbin/mysqld[0x52b279] /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x7c2)[0x52cbf2] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fa8bfd13f45] /usr/sbin/mysqld[0x51fd69] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
Json :
json file:
{
"cluster" : [
{
"id" : "nodel11_cluster",
"parameters" : {
"provision": {
"contrail_4": {
"enable_lbaas": true,
"global_config": {
"xmpp_auth_enable": true,
"xmpp_dns_auth_enable": true
},
"kernel_upgrade": false,
"ha": {
"contrail_external_vip": "10.204.217.34"
}
},
"openstack":{
"ha": {
"external_vip": "10.204.217.89",
"internal_vip": "10.204.217.89",
"external_virtual_router_id": "201"
},
"multi_tenancy": true,
"openstack_manage_amqp": true,
"keystone": {
"admin_password": "contrail123",
"admin_token": "admin"
}
}
}
}
}
],
"server": [
{
"cluster_id": "nodel11_cluster",
"domain": "englab.juniper.net",
"id": "nodel11-vm1",
"password": "c0ntrail123",
"network": {
"management_interface": "eth0",
"interfaces": [
{
"default_gateway": "10.204.217.254",
"dhcp": true,
"ip_address": "10.204.217.31/24",
"mac_address": "52:53:58:01:00:01",
"name": "eth0"
}
]
},
"roles": ["openstack", "contrail-controller", "contrail-analytics", "contrail-analyticsdb"]
},
{
"cluster_id": "nodel11_cluster",
"domain": "englab.juniper.net",
"id": "nodel11-vm2",
"password": "c0ntrail123",
"network": {
"management_interface": "eth0",
"interfaces": [
{
"default_gateway": "10.204.217.254",
"dhcp": true,
"ip_address": "10.204.217.32/24",
"mac_address": "52:53:58:01:00:02",
"name": "eth0"
}
]
},
"roles": ["openstack", "contrail-controller", "contrail-analytics", "contrail-analyticsdb"]
},
{
"cluster_id": "nodel11_cluster",
"domain": "englab.juniper.net",
"id": "nodel11-vm3",
"password": "c0ntrail123",
"network": {
"management_interface": "eth0",
"interfaces": [
{
"default_gateway": "10.204.217.254",
"dhcp": true,
"ip_address": "10.204.217.33/24",
"mac_address": "52:53:58:01:00:03",
"name": "eth0"
}
]
},
"roles": ["openstack", "contrail-controller", "contrail-analytics", "contrail-analyticsdb"]
},
{
"cluster_id": "nodel11_cluster",
"domain": "englab.juniper.net",
"id": "nodel11-vm4",
"password": "c0ntrail123",
"network": {
"management_interface": "eth0",
"interfaces": [
{
"default_gateway": "10.204.217.254",
"dhcp": true,
"ip_address": "10.204.217.34/24",
"mac_address": "52:53:58:01:00:04",
"name": "eth0"
}
]
},
"roles": ["contrail-lb"]
},
{
"cluster_id": "nodel11_cluster",
"domain": "englab.juniper.net",
"id": "nodel11-vm5",
"password": "c0ntrail123",
"network": {
"management_interface": "eth0",
"interfaces": [
{
"default_gateway": "10.204.217.254",
"dhcp": true,
"ip_address": "10.204.217.36/24",
"mac_address": "52:53:58:01:00:05",
"name": "eth0"
}
]
},
"roles": ["contrail-compute"]
}
],
"image": [
{
"category": "package",
"id": "regress_image",
"type": "contrail-ubuntu-package",
"path": "/root/contrail-cloud-docker_4.0.3.0-113-mitaka_trusty.tgz",
"version": "R4.1"
}
]
}
Logs and Json file copied at :
/cs-shared/bugs/<bug-id>/
