Fuel for OpenStack

Security vulnerability: OpenStack APIs and Horizon Web UI are prone to DOS attacks

Bug #1509986 reported by Adam Heczko on 2015-10-26

326

This bug affects 6 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Committed	Medium	Oleksiy Molchanov	Fuel for OpenStack 10.1
5.1.x	Won't Fix	Medium	MOS Maintenance	Fuel for OpenStack 5.1.1-updates
6.0.x	Won't Fix	Medium	MOS Maintenance	Fuel for OpenStack 6.0-updates
6.1.x	Won't Fix	Medium	MOS Maintenance	Fuel for OpenStack 6.1-updates
7.0.x	Won't Fix	Medium	MOS Maintenance	Fuel for OpenStack 7.0-updates
8.0.x	Won't Fix	Medium	MOS Maintenance	Fuel for OpenStack 8.0-updates
Mitaka	Fix Released	Medium	Oleksiy Molchanov	Fuel for OpenStack 9.2

Bug Description

Affected versions: MOS 7.0

It was observed that OpenStack APIs and Horizon are prone to DOS attack. Flooding http endpoints with large amount of malicious requests could lead to services malfunction.

Proposed solution:
Apply haproxy http rate request limiting. Consult scale team to develop appropriate limit values for various APIs, e.g. Glance image operations probably needs much lower values of http requests than Horizon.
Example haproxy configuration: https://github.com/dschneller/haproxy-http-based-rate-limiting/blob/master/haproxy.cfg

Explanation of that configuration:
https://blog.codecentric.de/en/2014/12/haproxy-http-header-rate-limiting/

See original description

Tags:

Adam Heczko (aheczko-mirantis) on 2015-10-26

description:

updated

Matthew Mosesohn (raytrac3r) on 2015-10-26

Changed in fuel:
milestone:	none → 8.0
assignee:	nobody → Fuel Library Team (fuel-library)
importance:	Undecided → Medium

Dmitry Pyzhov (dpyzhov) on 2015-10-27

tags:

added: area-library

Matthew Mosesohn (raytrac3r) on 2015-11-17

tags:

added: feature-security

Andrey Bubyr (abubyr) on 2015-11-17

description:

updated

Matthew Mosesohn (raytrac3r) on 2015-11-24

Changed in fuel:
status:	New → Confirmed

Revision history for this message

Matthew Mosesohn (raytrac3r) wrote on 2015-11-24:

@Georgy, can you please research and propose some haproxy limits for each OpenStack API? Doing limits on the haproxy load balancer is more effective than the DB-intensive limits in the APIs themselves.

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Georgy Okrokvertskhov (gokrokvertskhov)

Adam Heczko (aheczko-mirantis) on 2015-11-24

information type:

Private Security → Public Security

Stanislaw Bogatkin (sbogatkin) on 2015-12-14

tags:

added: team-bugfix

Dmitry Pyzhov (dpyzhov) on 2015-12-29

Changed in fuel:
milestone:	8.0 → 9.0

Revision history for this message

Mike Scherbakov (mihgen) wrote on 2016-03-10:

To implement haproxy config changes, recommendation are required from OpenStack team on those.

Changed in fuel:
assignee:	Georgy Okrokvertskhov (gokrokvertskhov) → MOS Puppet Team (mos-puppet)

Revision history for this message

Boris Bobrov (bbobrov) wrote on 2016-03-10:

I've marked 1555563 as a duplicate of this bugreport and set milestones from there.

Matthew Mosesohn (raytrac3r) on 2016-03-11

no longer affects:

fuel/future

Denis Klepikov (dklepikov) on 2016-03-11

tags:

added: customer-found

Revision history for this message

Dina Belova (dbelova) wrote on 2016-03-11:

Sorry, accidentally removed Newton series. Fuel drivers, please move it back.

no longer affects:	fuel/mitaka
no longer affects:	fuel/newton

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2016-03-11:

Confirmed for updates milestones, priority downgraded from Critical to Medium (see below for explanation)

This is not security vulnerability. Keystone endpoint is public in our default configuration, and as any service it has limited capacity and could be overloaded with number of requests that exceeds that capacity. If DoS attack is happening one could take appropriate measures - ban IPs the requests are coming from, filter specific types of packets, etc. There is nothing Keystone specific or OpenStack specific here.

Now this looks as feature request to put some limits on number of incoming requests. That should be discussed with product mgmt, scale and architecture groups. Then we could look into what could be done for older releases and existing deployments.

Revision history for this message

Alexey Stupnikov (astupnikov) wrote on 2016-03-18:

Set this bug's status to Won't Fix (wontfix-munotapplic) for 5.1 and 6.0 branch since we don't deliver Fuel fixes in maintenance updates for 5.1.1 and 6.0.

tags:

added: wontfix-munotapplic

Revision history for this message

Bug Checker Bot (bug-checker) wrote on 2016-03-28: Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags:

added: need-info

Revision history for this message

Ivan Berezovskiy (iberezovskiy) wrote on 2016-04-06:

It's too late to fix medium bug in 9.0

Dmitry Pyzhov (dpyzhov) on 2016-04-13

no longer affects:	fuel/newton
Changed in fuel:
milestone:	9.0 → 10.0

Revision history for this message

Adam Heczko (aheczko-mirantis) wrote on 2016-06-23:

Proposed approach is described here:
https://javapipe.com/iptables-ddos-protection

Maksim Malchuk (mmalchuk) on 2016-06-29

tags:

removed: area-library

Denis Egorenko (degorenko) on 2016-07-11

tags:

added: 10.0-reviewed

Ivan Berezovskiy (iberezovskiy) on 2016-07-25

Changed in fuel:
assignee:	Max Yatsenko (myatsenko) → Fuel Sustaining (fuel-sustaining-team)

Revision history for this message

Rodion Tikunov (rtikunov) wrote on 2016-08-18:

#10

Closing this bug as Won't Fix for MOS 6.1 as we have finished active support for those releases and don't merge non-critical and non-security fixes there anymore.

Revision history for this message

Denis Meltsaykin (dmeltsaykin) wrote on 2016-09-02:

#11

Closing as Won't Fix for 8.0 as this is a medium importance arguable feature-request.

Oleksiy Molchanov (omolchanov) on 2016-10-06

Changed in fuel:
assignee:	Fuel Sustaining (fuel-sustaining-team) → Oleksiy Molchanov (omolchanov)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-10-11: Fix proposed to fuel-library (master)

#12

Fix proposed to branch: master
Review: https://review.openstack.org/384994

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-03: Fix merged to fuel-library (master)

#13

Reviewed: https://review.openstack.org/384994
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0fb99f1ca7fe0925051060d552368bde31734c9e
Submitter: Jenkins
Branch: master

commit 0fb99f1ca7fe0925051060d552368bde31734c9e
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Oct 24 16:23:12 2016 +0300

Add few ddos protection rules to iptables

Change-Id: I771cfdf7db9acd0c4de7fba8b775f66166c0b461
Partial-Bug: 1509986

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-18: Fix proposed to fuel-library (stable/mitaka)

#14

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/399497

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-18: Fix proposed to fuel-library (stable/newton)

#15

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/399498

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-23: Fix merged to fuel-library (stable/newton)

#16

Reviewed: https://review.openstack.org/399498
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=607701ab9ad5595ac5aa7a631c37b91285555b01
Submitter: Jenkins
Branch: stable/newton

commit 607701ab9ad5595ac5aa7a631c37b91285555b01
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Oct 24 16:23:12 2016 +0300

Add few ddos protection rules to iptables

    Change-Id: I771cfdf7db9acd0c4de7fba8b775f66166c0b461
    Partial-Bug: 1509986
    (cherry picked from commit 0fb99f1ca7fe0925051060d552368bde31734c9e)

tags:	added: in-stable-newton
tags:	added: in-stable-mitaka

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-23: Fix merged to fuel-library (stable/mitaka)

#17

Reviewed: https://review.openstack.org/399497
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3c3fa05c3d1725d00c62ab7ba3fcb97c67b4a0b7
Submitter: Jenkins
Branch: stable/mitaka

commit 3c3fa05c3d1725d00c62ab7ba3fcb97c67b4a0b7
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Oct 24 16:23:12 2016 +0300

Add few ddos protection rules to iptables

    Change-Id: I771cfdf7db9acd0c4de7fba8b775f66166c0b461
    Partial-Bug: 1509986
    (cherry picked from commit 0fb99f1ca7fe0925051060d552368bde31734c9e)

Revision history for this message

Dmitry Pyzhov (dpyzhov) wrote on 2016-12-13:

#18

All patches are merged.

Changed in fuel:
status:	In Progress → Fix Committed
milestone:	10.0 → 10.1

Revision history for this message

Ilya Bumarskov (ibumarskov) wrote on 2017-01-30:

#19

Verified on snapshot-id #822
Firewall rules were apllied on all nodes (include nodes with compute and cinder roles) in cluster:

root@node-2:~# iptables --list | grep block
DROP all -- anywhere anywhere /* 010 block invalid packets */ ctstate INVALID
DROP tcp -- anywhere anywhere tcp flags:!FIN,SYN,RST,ACK/SYN /* 020 block not-syn new packets */ ctstate NEW
DROP tcp -- anywhere anywhere /* 030 block uncommon mss values */ ctstate NEW tcpmss match !536:65535
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE /* 040 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN/FIN,SYN /* 050 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:SYN,RST/SYN,RST /* 060 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN/FIN,SYN /* 070 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,RST/FIN,RST /* 080 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,ACK/FIN /* 090 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:ACK,URG/URG /* 100 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,ACK/FIN /* 110 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:PSH,ACK/PSH /* 120 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,PSH,ACK,URG /* 130 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE /* 140 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,PSH,URG /* 150 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,PSH,URG /* 160 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,ACK,URG /* 170 block packets with bogus tcp flags */

Verified on snapshot-id #822
Firewall rules were apllied on all nodes (include nodes with compute and cinder roles) in cluster:

root@node-2:~# iptables --list | grep block
DROP       all  --  anywhere             anywhere             /* 010 block invalid packets */ ctstate INVALID
DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN /* 020 block not-syn new packets */ ctstate NEW
DROP       tcp  --  anywhere             anywhere             /* 030 block uncommon mss values */ ctstate NEW tcpmss match !536:65535
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE /* 040 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN/FIN,SYN /* 050 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:SYN,RST/SYN,RST /* 060 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN/FIN,SYN /* 070 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,RST/FIN,RST /* 080 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,ACK/FIN /* 090 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:ACK,URG/URG /* 100 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,ACK/FIN /* 110 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:PSH,ACK/PSH /* 120 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,PSH,ACK,URG /* 130 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE /* 140 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,PSH,URG /* 150 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,PSH,URG /* 160 block packets with bogus tcp flags */
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,ACK,URG /* 170 block packets with bogus tcp flags */