Fuel for OpenStack

[vbox] No Internet connection on compute nodes in HA mode when run under VBox

Bug #1288135 reported by Timur Nurlygayanov
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Medium
Serhii Ovsianikov
Fuel for OpenStack 6.1

Bug Description

Steps To Reproduce:
1. Deploy Fuel 4.1 using scripts for Virtual Box
2. Deploy OpenStack with HA mode, Cent OS, 3 controller and 1 compute, Neutron with VLANs (and install Murano, Savanna and Ceilometer).
3. Login to compute node and ping mail.ru: ping mail.ru
4. Create VM in OpenStack and ping 8.8.8.8 or mail.ru from this host

Observed Result:
We can not ping any hosts in Internet from compute nodes and VMs.
We can see also that name resolving works good (wen we ping mail.ru we know th IP address of this host)

[root@node-2 ~]# ping mail.ru
PING mail.ru (217.69.139.199) 56(84) bytes of data.
^C
--- mail.ru ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2936ms

[root@node-2 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
domain domain.tld
search domain.tld
nameserver 10.20.0.2

[root@node-2 ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.20.0.0 * 255.255.255.0 U 0 0 0 br-fw-admin
192.168.1.0 * 255.255.255.0 U 0 0 0 br-storage
172.16.0.0 * 255.255.255.0 U 0 0 0 br-ex
192.168.0.0 * 255.255.255.0 U 0 0 0 br-mgmt
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
link-local * 255.255.0.0 U 1002 0 0 eth0
link-local * 255.255.0.0 U 1003 0 0 eth1
link-local * 255.255.0.0 U 1004 0 0 eth2
link-local * 255.255.0.0 U 1009 0 0 br-ex
link-local * 255.255.0.0 U 1010 0 0 br-mgmt
link-local * 255.255.0.0 U 1011 0 0 br-storage
link-local * 255.255.0.0 U 1012 0 0 br-fw-admin
link-local * 255.255.0.0 U 1013 0 0 br-prv
default 172.16.0.1 0.0.0.0 UG 0 0 0 br-ex

[root@node-2 ~]# tracepath mail.ru
 1: 172.16.0.6 (172.16.0.6) 0.185ms pmtu 1500
 1: 172.16.0.1 (172.16.0.1) 1.498ms
 1: 172.16.0.1 (172.16.0.1) 0.305ms
 2: no reply
 3: no reply
 4: no reply
 5: no reply

See original description
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

10.20.0.2 in this case it is private IP of Fuel Master node.

Changed in fuel:
importance: Undecided → High
assignee: nobody → Vladimir Kuklin (vkuklin)
milestone: none → 4.1.1
status: New → Confirmed
status: Confirmed → New
description: updated
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Steps How We can fix this problem:

in Virtual Box need to change the type of network to NAT and execute on the host machine (where Virtual Box is running) the following command:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

where eth0 - the interface with the access to the Internet.

Vladimir Kuklin (vkuklin)
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Miroslav Anashkin (manashkin)
Evgeniy L (rustyrobot)
Changed in fuel:
status: New → Confirmed
Vladimir Kuklin (vkuklin)
Changed in fuel:
milestone: 4.1.1 → 5.0
tags: added: backports-4.1.1
Revision history for this message
Mike Scherbakov (mihgen) wrote : Re: No Internet connection on compute nodes in HA mode when run under VBox

Lowering priority: issue exists only if you run setup under VirtualBox scripts due to configuration of networking on the host machine.
Suggested
> iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
 obviously won't work for Mac users.

Changed in fuel:
importance: High → Medium
summary: - No Internet connection on compute nodes in HA mode
+ No Internet connection on compute nodes in HA mode when run under VBox
Mike Scherbakov (mihgen)
Changed in fuel:
milestone: 5.0 → 5.1
Mike Scherbakov (mihgen)
Changed in fuel:
assignee: Miroslav Anashkin (manashkin) → Timur Nurlygayanov (tnurlygayanov)
milestone: 5.1 → 5.0
Mike Scherbakov (mihgen)
Changed in fuel:
milestone: 5.0 → 5.1
Timur Nurlygayanov (tnurlygayanov)
Changed in fuel:
assignee: Timur Nurlygayanov (tnurlygayanov) → Miroslav Anashkin (manashkin)
Dmitry Ilyin (idv1985)
summary: - No Internet connection on compute nodes in HA mode when run under VBox
+ [vbox] No Internet connection on compute nodes in HA mode when run under
+ VBox
Miroslav Anashkin (manashkin)
Changed in fuel:
status: Confirmed → Triaged
Dmitry Borodaenko (angdraug)
Changed in fuel:
milestone: 5.1 → 6.0
Revision history for this message
Miroslav Anashkin (manashkin) wrote :

Working network settings for MacOS overview available here:
https://drive.google.com/file/d/0BwNB1Fj6mQMBRnJHYjBtSXlIajQ/edit?usp=sharing

To set up NAT for VBox host-only adapter under MacOS one may do the following (all under root account!)
1.
`sysctl -w net.inet.ip.forwarding=1`

2. Do not use "ipfw" — use "pf" only.
Below is the pf config example with the necessary rules declared. Simply add the similar settings to your pf.conf:

`# cat /etc/pf.conf`
#######################
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
nat on en0 inet from ! (en0) to any -> (en0)
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
pass in on vboxnet0
pass in on vboxnet1
pass in on vboxnet2
pass in on vboxnet3
pass in on vboxnet4
pass in on vboxnet5
pass in on vboxnet6
pass in on vboxnet7
pass in on vboxnet8
pass in on vboxnet9
pass in on vboxnet10
pass in on vboxnet11
pass in on vboxnet12
pass in on vboxnet13
pass in on vboxnet14
pass in on vboxnet15
#######################

3. Apply the settings:
`pfctl -f /etc/pf.conf`

The key rules in the settings listed above are `nat on en0 inet ...`
and all that "pass" rules, all the remained rules are mostly defaults.

Miroslav Anashkin (manashkin)
Changed in fuel:
assignee: Miroslav Anashkin (manashkin) → Fabrizio Soppelsa (fsoppelsa)
Fabrizio Soppelsa (fsoppelsa)
Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Fabrizio Soppelsa (fsoppelsa) wrote :

Some minor issues on setting up /etc/pf.conf: our rules must be in precise order, otherwise they won't be accepted by `pfctl`.

In /etc/pf.conf there's already a minimal configuration, on brand-new installations:

scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
dummynet-anchor "com.apple/*"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"

For instance, one should inject

nat on en0 inet from ! (en0) to any -> (en0)

Between ^dummynet and ^anchor

Keeping in mind that this file won't be likely modified by the desktop user and probably he won't try Fuel on production (with production defined /etc/pf.conf), would you suggest to simply backup /etc/pf.conf to /etc/pf.conf.bak, reload PF rules, run the Fuel demo and finally restore /etc/pf.conf.bak in place?

Other suggestions work like a charm.

Also, solving this bug means that ./launch*.sh must be run by root.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (master)

Fix proposed to branch: master
Review: https://review.openstack.org/119717

Fabrizio Soppelsa (fsoppelsa)
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/120429

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-main (master)

Change abandoned by Fabrizio Soppelsa (<email address hidden>) on branch: master
Review: https://review.openstack.org/120429
Reason: Duplicate

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Fabrizio Soppelsa (<email address hidden>) on branch: master
Review: https://review.openstack.org/119717
Reason: Duplicate

Fabrizio Soppelsa (fsoppelsa)
Changed in fuel:
status: Fix Committed → In Progress
Roman Prykhodchenko (romcheg)
Changed in fuel:
milestone: 6.0 → 6.1
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Fabrizio Soppelsa (fsoppelsa) → Serhiy Ovsianikov (sovsianikov)
Vladimir Kuklin (vkuklin)
tags: added: non-release
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Fabrizio Soppelsa (<email address hidden>) on branch: master
Review: https://review.openstack.org/119717
Reason: Merge conflict

Revision history for this message
Serhii Ovsianikov (sovsianikov) wrote :

There is similar bug https://bugs.launchpad.net/fuel/+bug/1442988 and review https://review.openstack.org/#/c/173768

Changed in fuel:
status: In Progress → Fix Released
Serhii Ovsianikov (sovsianikov)
Changed in fuel:
status: Fix Released → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/119717
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=deda3b648409aeba6ceada212f2c27764fc1915d
Submitter: Jenkins
Branch: master

commit deda3b648409aeba6ceada212f2c27764fc1915d
Author: Serhiy Ovsianikov <email address hidden>
Date: Thu May 14 20:20:18 2015 +0300

    Fix for VBox "No Internet connection on compute nodes"

    The scripts work without any firewall tuning on the user's computers.
    The Fuel master node is forwarding the public network through
    the existing NAT interface.

    Change-Id: I2962bccb39828da4ff011889f990a67d89322ca3
    Closes-Bug: #1288135

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.