Rsyslogd stores all remote logs in '172.17.42.1' folder

Bug #1332517 reported by Artem Panchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Bogdan Dobrelya
5.0.x
Fix Committed
High
Bogdan Dobrelya

Bug Description

api: '1.0'
astute_sha: 694b5a55695e01e1c42185bfac9cc7a641a9bd48
build_id: 2014-06-20_00-31-14
build_number: '262'
fuellib_sha: 25eb618a33a2ec87bc56f6bad16dc25b1837f0f0
fuelmain_sha: 4f547561532baf5f26733bf66db692dc5b61806d
mirantis: 'yes'
nailgun_sha: 0c5e3b94fdd6bc9a50d5f840bf5151f95a23d908
ostf_sha: 2f30e5cab5bec1f1e2fd80e26e4da771a8ffe2d4
production: docker
release: '5.1'

Steps to reproduce:

 1. Deploy master node, boot slaves
 2. Go to 'Logs' tab and check logs from slave nodes

Expected:

 - got the list of bootstrap logs and their content

Actual result:

 - remote logs aren't available

Also, remote logs don't appear after successful cluster deployment. The reason of this issue is iptables rule added to the top of 'POSTROUTING' chain:

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 4 226 MASQUERADE all -- * * 10.108.180.0/24 0.0.0.0/0

It causes masquerading of all packets sent to rsyslog container, so it stores all logs to the /var/log/remote/172.17.42.1/ (172.17.42.1 master node IP on docker0 interface). I was able to fix the issue remote with logs by removing that rule and restarting rsyslog container.

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :
Changed in fuel:
status: New → In Progress
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

The source of the issue is that the rule from ./deployment/puppet/nailgun/manifests/iptables.pp:
$rule = "-s $network -j MASQUERADE"
command => "iptables -t nat -I POSTROUTING 1 $rule; \

wants to be the very first while the Fuel master node is deploying. That rule is needed in order to provide the routing options for OSt nodes via the Fuel master node.

But for some cases (like this one in the ticket), the docker containers startup process could place its own "very first" rules ahead,
./deployment/puppet/docker/templates/functions.sh.erb:
iptables -t nat -I POSTROUTING 1 -s "$NETWORK/$PREFIX" -p $proto -m $proto --dport $port -j ACCEPT

As a result we'd have result in the broken NAT chain then we've got an admin network masquerading rule have been placed before the docker rules.

The solution is to add an admin network masquerading rule with the -A option instead of -I 1. That would do the trick unless we have placed the blocking rules at the end of the NAT list. We don't use any for now, so it is OK to do so...

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Or it is better to update the admin network masquerading rule by adding ' ! -o docker0' :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/101519

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Why is the priority High for 5.1 and Medium for 5.0. Is impact of this bug in 5.0 less significant?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/101519
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=221df8c41943a093c7495178a30740aa3a543426
Submitter: Jenkins
Branch: master

commit 221df8c41943a093c7495178a30740aa3a543426
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Jun 20 15:57:42 2014 +0300

    Exclude docker from the admin net masquerading

    We should exclude -s docker0 form the nailgun iptables
    rule for admin network masquerading in order to
    elliminate the potential issues related with race condition
    during master node deployment.

    Closes-bug: #1332517

    Change-Id: I942da109d83cce65880a3f27cb1d5301d300da50
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/101912

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.0)

Reviewed: https://review.openstack.org/101912
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0f7e3d0c59f9836966a6c1f2f69cdff163e93d65
Submitter: Jenkins
Branch: stable/5.0

commit 0f7e3d0c59f9836966a6c1f2f69cdff163e93d65
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Jun 20 15:57:42 2014 +0300

    Exclude docker from the admin net masquerading

    We should exclude -s docker0 form the nailgun iptables
    rule for admin network masquerading in order to
    elliminate the potential issues related with race condition
    during master node deployment.

    Closes-bug: #1332517

    Change-Id: I942da109d83cce65880a3f27cb1d5301d300da50
    Signed-off-by: Bogdan Dobrelya <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.