Public VIP is inaccessible from external networks

Bug #1431547 reported by Victor Ryzhenkin
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Vladimir Kuklin
5.0.x
Invalid
Critical
Fuel QA Team
5.1.x
Invalid
Critical
Fuel QA Team
6.0.x
Invalid
Critical
Fuel QA Team
6.1.x
Fix Released
Critical
Vladimir Kuklin

Bug Description

ISO: {"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-11_21-47-59", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "187", "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "a720a2da99690eb2d2c19ddc5d739384312a8ac2", "production": "docker", "python-fuelclient_sha": "0f4ca9c2798da34797dd082130d22cac04c998a9", "astute_sha": "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0791400dd8224647ff9a5cb8051ce82b2c8863b1", "fuellib_sha": "cfdfcbdb0197f606b4c93e6dd4011525df9a3ff8"}

STEPS:
1. Deploy OS-neutron GRE, HA (3 Virtual Controllers, 1 Virtual Compute)
2. Log into Horizon.

Expected:
Horizon UI accesible via external network

Actual result:
Horizon UI accesible only from controller nodes.

NOTES:
I can't find any logs with failures of network, but in one case when i executed "pcs resource",
resource vip__public was in FAILURE state.
In Iptables i can't find any information about ip of haproxy, but resource vip__public should provide this information to iptables.

I tried to upload snapshot, but to my amazement it was over 200M. I'll take it on my Dropbox later.

description: updated
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :
Changed in fuel:
importance: Undecided → Critical
assignee: nobody → Fuel Library Team (fuel-library)
status: New → Confirmed
milestone: none → 6.1
Revision history for this message
Stanislav Makar (smakar) wrote :

Could you please tell us how you deployed this env (e.g. manually, using dos.py, or virtualbox scripts )?

Revision history for this message
Stanislav Makar (smakar) wrote :

related bug: https://bugs.launchpad.net/fuel/+bug/1396126

looks like it is due to above bug

Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Stanislav, this env was deployed manually.

More information:
We create ssh tunnel to access Horizon.
Our public router was alive.
We deployed instance and we has access to this instance from external network and it was succesfully pinged.
Instance was in the same network with controllers.

Also, for debugging, we shutdown our controllers, and create test VMs. We have configured networking on them, and boot to bootstrap.
After that, we configure public network on VMs and add ip from every controller to this VMs. Every VM was successfully pinged from our external network.

Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Stanistav, in our case deployment was finished successfully.. Not failed.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

I lowered the bug priority as it doesn't affect the deployment, please update the status back if you think it should be higher

Changed in fuel:
importance: Critical → Medium
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Bogdan, i'm think that this bug should be with high priority at least.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Bogdan, if Horizon dashboard is not available after the deployment it is Critical bug because we can not release the product with this issue.

Importance changed to High, we need to track this issue and fix it as soon as possible, it is blocker for QA team and for release.

Changed in fuel:
importance: Medium → High
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

QA team, please confirm either this issue affects other releases

description: updated
description: updated
description: updated
description: updated
Revision history for this message
Yaroslav Lobankov (ylobankov) wrote :

I have never encountered this issue in 6.0.1. I have deployed the latest ISO (#133) . The issue has not been reproduced.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

The RC of this issue is found here https://bugs.launchpad.net/mos/+bug/1434088

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

This issue should be related to the 6.1 specific changes which came with the networking refactoring

summary: - Horizon dashboard unavailable after env deploying complete
+ Horizon dashboard unavailable after env deploying complete due to Public
+ VIP is inaccessible from external networks
Revision history for this message
Bogdan Dobrelya (bogdando) wrote : Re: Horizon dashboard unavailable after env deploying complete due to Public VIP is inaccessible from external networks

Sergey Kolekonov (skolekonov) wrote 13 minutes ago:
I've also found out that Attribute "gateway" for Pacemaker resource vip__public is empty, but it should contain the appropriare gateway.
I've just executed:
pcs resource update vip__public gateway=<my gateway>
and vip is available from the outside (including Horizon)

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

I'm not sure either replacing 240.0.0.1 route to gateway IP is an appropriate solution, but it works. Also I noticed that affected environments do not have a scope link route inside of haproxy namespace, which is:

default dev hapr-p scope link metric 10

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/166273

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote : Re: Horizon dashboard unavailable after env deploying complete due to Public VIP is inaccessible from external networks

Please try the latest ISO.

Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Still reproduced with latest iso.

Revision history for this message
Vadim Rovachev (vrovachev) wrote :

Patch and bug verified on:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "233"
  build_id: "2015-03-26_21-32-43"
  nailgun_sha: "b163f6fc77d6639aaffd9dd992e1ad96951c3bbf"
  python-fuelclient_sha: "e5e8389d8d481561a4d7107a99daae07c6ec5177"
  astute_sha: "3f1ece0318e5e93eaf48802fefabf512ca1dce40"
  fuellib_sha: "9c7716bc2ce6075065d7d9dcf96f4c94662c0b56"
  ostf_sha: "a4cf5f218c6aea98105b10c97a4aed8115c15867"
  fuelmain_sha: "320b5f46fc1b2798f9e86ed7df51d3bda1686c10"

Without patch bug reproduced.
With patch bug not reproduced.

Patch works.

Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Vladimir Kuklin (vkuklin)
summary: - Horizon dashboard unavailable after env deploying complete due to Public
- VIP is inaccessible from external networks
+ Public VIP is inaccessible from external networks
Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/166273
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8e76bdeb583fac6ee7b3769b2754831dcff871e6
Submitter: Jenkins
Branch: master

commit 8e76bdeb583fac6ee7b3769b2754831dcff871e6
Author: Vladimir Kuklin <email address hidden>
Date: Fri Apr 3 02:50:16 2015 +0300

    Revert of gateways for VIPs

    This commit sets default route for public interfaces
    and also sets static routes for haproxy namespaces to
    the networks which are connected to the host directly
    with metric of 10000 to ensure that these routes do
    not override routes installed by kernel on interfaces
    configuration stage

    Regression caused by fix of
    https://bugs.launchpad.net/fuel/+bug/1433150

    Change-Id: I7df8efad4aa119896314e77c29d9b024599dc865
    Closes-Bug: 1431547

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Vadim Rovachev (vrovachev) wrote :

verified on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "281"
  build_id: "2015-04-06_03-15-13"
  nailgun_sha: "a2dc04af9c5838a990adeab81789d4c090943701"
  python-fuelclient_sha: "9ad42671130deb2cc81135fb453ef0793883be2c"
  astute_sha: "240bfefe0a0b5af21b79854d0c96d8a3ee0fd885"
  fuellib_sha: "601cc8e1dccbdcf328f83f071270609e824d6c5a"
  ostf_sha: "b978badb43243d3f1b85dde15b8e301a21f12254"
  fuelmain_sha: "f90d7d40ceff39ee6d7587b183c63bd00ed0743f"

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.