Fuel for OpenStack

adding nodes after the deplpoyment finished fails, because some active hosts are already declared (or appears to be deleted)

Bug #1704157 reported by Zoltan Langi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Incomplete
Undecided
MOS Maintenance
Fuel for OpenStack 11.x-updates

Bug Description

Detailed bug description:

Installed openstack Ocata with fuel 11, installation finished. After a few hours of finishing the install, I've added more compute nodes, that succeeded.
After a few days I wanted to add even more compute nodes, but the instalation fails with the following message:

Error
All nodes are finished. Failed tasks: Task[hosts/1], Task[hosts/3], Task[hosts/2] Stopping the deployment process!

In the puppet.log on the nodes, found this:

2017-07-13 14:49:30 +0000 Puppet (err): Duplicate declaration: Host[node-13.xx.com] is already declared; cannot redeclare at /etc/puppet/modules/osnailyfacter/manifests/hosts/hosts.pp:25 on node node-1.xx.com
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:500:in `fail_on_duplicate_type_and_title'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:86:in `add_one_resource'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:72:in `block in add_resource'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:71:in `each'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:71:in `add_resource'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:88:in `add_resource'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:59:in `block (3 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:64:in `exceptwrap'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:43:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions/create_resources.rb:73:in `block in <top (required)>'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:164:in `block (2 levels) in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:157:in `block in newfunction'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:43:in `block (2 levels) in <top (required)>'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:36:in `each'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/ensure_resource.rb:36:in `block in <top (required)>'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:164:in `block (2 levels) in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:157:in `block in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/function.rb:42:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/ifstatement.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/type.rb:129:in `evaluate_code'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:77:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:236:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:236:in `evaluate_classes'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:60:in `block (3 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:64:in `exceptwrap'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:43:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:42:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:12:in `block in each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:11:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/branch.rb:11:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `collect'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/resource.rb:25:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/type.rb:129:in `evaluate_code'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:77:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:426:in `evaluate_main'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:121:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:34:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:95:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:93:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/2.3.0/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:92:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:201:in `find'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:222:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'

Checking the source code, found this file:

root@node-1:~# cat /etc/hiera/deleted_nodes.yaml
deleted_nodes:
- node-13.xx.com
- node-31.xx.com
- node-30.xx.com
- node-11.xx.com
- node-35.xx.com
- node-17.xx.com
- node-16.xx.com
- node-10.xx.com

According to this file, these nodes are deleted but that's not true because they're online and running, serving right now, fuel node list shows them as online.

Impact:
Currently I can't add or remove any nodes from my environment or do any changed like password, certificate change because it fails on this error.

Version: fuel 11 ocata

Thank you for looking into it in advance,
Zoltan

See original description
Zoltan Langi (zlangi)
description: updated
Zoltan Langi (zlangi)
summary: - adding compute nodes after deplpoyment finished failes because some
- hosts are already declared
+ adding nodes after deplpoyment finished fails because some hosts are
+ already declared (or appears to be deleted)
summary: - adding nodes after deplpoyment finished fails because some hosts are
- already declared (or appears to be deleted)
+ adding nodes after the deplpoyment finished fails, because some active
+ hosts are already declared (or appears to be deleted)
description: updated
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Please provide diagnostic snapshot.

Changed in fuel:
status: New → Incomplete
milestone: none → 11.x-updates
assignee: nobody → MOS Maintenance (mos-maintenance)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.