Keystone's IANA-assigned default port in linux local ephemeral port range: [Errno 98] Address already in use

Given that 35357 is keystone's IANA assigned port, I don't think it's reasonable to expect keystone to change ports in the short term. In the long term (after dropping API v2 support), keystone will be deployable on any one port. Using 35357 theoretically aids discoverability a tiny bit, but there are other means of discoverability that we can investigate in the future (such as RFC 2782).

Missed the documentation requirement in the last paragraph. 100% agree!

Fix proposed to branch: master
Review: https://review.openstack.org/57577

In keystone and openstack-manuals, I think we should add documentation recommending that deployers explicitly exclude port 35357 from the available ephemeral ports, regardless of what the ephemeral port range (ip_local_port_range ) is configured to be. In a clean linux environment, this would be as simple as:

    echo 35357 > /proc/sys/net/ipv4/ip_local_reserved_ports

Fix proposed to branch: master
Review: https://review.openstack.org/58013

This should go in the Configuration Reference, Operations Guide, and Cloud Admin Guide, preferably in a common location.

Ignore the details of comment #4 above -- a better solution is developing in the above code reviews.

Fix proposed to branch: master
Review: https://review.openstack.org/59528

Doesn't show up often but it does show up:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwia2V5c3RvbmUtYWxsIHNlcnZlIEZhaWxlZCB0byBzdGFydCB0aGUgYWRtaW4gc2VydmVyXCIgQU5EIGZpbGVuYW1lOlwibG9ncy9zY3JlZW4ta2V5LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg2NjE2NDQ4ODU5fQ==

Pushed a patch to track this in elastic-recheck: https://review.openstack.org/#/c/60917/

Matt, Thanks for adding that!

so we just got a failure spike on this, anyone know why?

The spike may be related to jenkins01 crashing. If it reused any slave nodes when it came back up (because nodepool didn't get the appropriate events due to jenkins crashing) then keystone could have been running on those nodes preventing it from starting when the second jobs ran.

Is there a ci bug for the jenkins crash? Because right now we are e-r assigning the jenkins crash and slave reuse to this bug, which is really not the right thing to do.

In openstack-manuals, this will now be dealt holistically across all services by this bug : https://bugs.launchpad.net/openstack-manuals/+bug/1261617 " Provide information and instructions on default ports and firewall rules "

We are seeing this in the gate again

Adam young suggested setting KEYSTONE_AUTH_PORT=32635 in gate

Any non-claimed port will work: look in /etc/services

We want to make sure it is not a port used by openstack. So that rules out the ports for LDAP, MySQL, PostGresql, Memcached, and then any of the openstack services. These should all be registered in /etc/services.

There are many unclaimed ports under 32768. What is essential is that the machines consider the port non-ephemeral, and it not conflict.

 looks like there is an open range from
iceedcp_rx 31949
to
iracinghelper 32034 so

how about 32999.

This is just for the gate, not a change we are going to make to devstack proper or Keystone

This bug has a very consistent rate of occurrence, but the elastic-recheck query for this bug was a bit too vague, and picked up several false positives. Eliminating the false positives in the query reduced the 7 day impact of this bug from 15 hits to just 6: https://review.openstack.org/69997

Note: logs from keystone meeting http://eavesdrop.openstack.org/meetings/keystone/2014/keystone.2014-01-21-18.01.log.html

Keystone docs should be revised to include the caveats around usage of port 35357 in production, along with some pointers to suggested remedies.

Reviewed: https://review.openstack.org/57577
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=6c57fbab26e40af5c5b19b46fb3da39341f34dab
Submitter: Jenkins
Branch: master

commit 6c57fbab26e40af5c5b19b46fb3da39341f34dab
Author: Clark Boylan <email address hidden>
Date: Wed Nov 20 17:00:21 2013 -0800

    Set keystone admin_bind_host to KEYSTONE_SERVICE_HOST

    On Linux ports 32768-61000 can be used by just about
    anything needing a socket. Keystone's IANA assigned port is 35357.
    Occasionally something else will be using port 35357 first because Linux
    allows this. Workaround is to bind to port 127.0.0.1 instead of 0.0.0.0.
    $KEYSTONE_SERVICE_HOST gets its value from $SERVICE_HOST which is set to
    127.0.0.1 in the gate.

    "Ephemeral (client) ports will *never* be sourced from 0.0.0.0, and are
    uniquely identified by the full connection five-tuple (proto, src IP,
    src port, dst IP, dst port) anyway, allowing them to overlap src IP/src
    port as long as proto/dst IP/dst port are different. Thus it is up to
    keystone/devstack to bind more appropriately and not use wildcard bind
    addresses unless explicitly necessary for some reason. For example, in
    the log output, the URLs are configured with dst IPs of 127.0.0.1
    anyway, so binding explicitly to localhost would change nothing, while
    skirting this particular edge case nicely." ~Evan Callicoat

    This doesn't fix bug 1253482 it works around it while a better solution
    is prepared (running keystone behind apache in devstack).

    Co-Authored-By: Joe Gordon <email address hidden>
    Change-Id: I112309661dadf8b753c3311182f82464d9d3595e
    Related-bug: #1253482

The spike in today's elastic rechecks against this bug are actually instances of bug 1277507, and a result of the elastic recheck query as described in https://bugs.launchpad.net/devstack/+bug/1253482/comments/19

This remains a gate reset issue. So I'm making this critical for keystone. There needs to be some better retry logic around this or some where to make this this not continue to be a race in the gate.

It definitely seems to still not be fixed in stable/havana either, since it's breaking grenade jobs during the setup phase:

(keystone.token.provider): 2014-03-14 12:08:46,033 WARNING provider get_token_provider keystone.conf [signing] token_format is deprecated in favor of keystone.conf [token] provider
(keystone.common.environment.eventlet_server): 2014-03-14 12:08:46,128 INFO eventlet_server start Starting /opt/stack/old/keystone/bin/keystone-all on 0.0.0.0:35357
(keystone): 2014-03-14 12:08:46,128 CRITICAL log logging_excepthook [Errno 98] Address already in use
key failed to start

http://logs.openstack.org/76/80476/1/gate/gate-grenade-dsvm/ae07834/logs/old/screen-key.txt.gz?level=INFO

Looks like our e-r query isn't working in all cases anymore:

http://logs.openstack.org/15/84215/4/gate/gate-tempest-dsvm-neutron/e82ae58/logs/screen-key.txt.gz

At least I didn't get any message from e-r in this patch: https://review.openstack.org/#/c/84215

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/93641

Reviewed: https://review.openstack.org/93641
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=e4dda729ef9670927978f49c9d21f160c0457efb
Submitter: Jenkins
Branch: stable/havana

commit e4dda729ef9670927978f49c9d21f160c0457efb
Author: Clark Boylan <email address hidden>
Date: Wed Nov 20 17:00:21 2013 -0800

    Set keystone admin_bind_host to KEYSTONE_SERVICE_HOST

    NOTE: squashed with follow-up Make admin_bind_host configurable
    to avoid regression bug #1283803

    On Linux ports 32768-61000 can be used by just about
    anything needing a socket. Keystone's IANA assigned port is 35357.
    Occasionally something else will be using port 35357 first because Linux
    allows this. Workaround is to bind to port 127.0.0.1 instead of 0.0.0.0.
    $KEYSTONE_SERVICE_HOST gets its value from $SERVICE_HOST which is set to
    127.0.0.1 in the gate.

    "Ephemeral (client) ports will *never* be sourced from 0.0.0.0, and are
    uniquely identified by the full connection five-tuple (proto, src IP,
    src port, dst IP, dst port) anyway, allowing them to overlap src IP/src
    port as long as proto/dst IP/dst port are different. Thus it is up to
    keystone/devstack to bind more appropriately and not use wildcard bind
    addresses unless explicitly necessary for some reason. For example, in
    the log output, the URLs are configured with dst IPs of 127.0.0.1
    anyway, so binding explicitly to localhost would change nothing, while
    skirting this particular edge case nicely." ~Evan Callicoat

    This doesn't fix bug 1253482 it works around it while a better solution
    is prepared (running keystone behind apache in devstack).

    Co-Authored-By: Joe Gordon <email address hidden>
    Change-Id: I112309661dadf8b753c3311182f82464d9d3595e
    Related-bug: #1253482
    (cherry picked from commit 6c57fbab26e40af5c5b19b46fb3da39341f34dab)
    (cherry picked from commit 041fa712472d887550a540dd50ade546f847c6b4)

The keystone port can be reserved quite easily in linux via:

echo "35357" | sudo cat /proc/sys/net/ipv4/ip_local_reserved_ports

@Vish,

That has been commented as a "fix" but was disliked for a number of reasons. I agree it is the easiest fix. (Unfortunately, IANA and linux are at odds here and we're losing because of it).

Reviewed: https://review.openstack.org/58013
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1b338ab0bf8cbaef03f59d7d3a474ccbed2c9029
Submitter: Jenkins
Branch: master

commit 1b338ab0bf8cbaef03f59d7d3a474ccbed2c9029
Author: Dolph Mathews <email address hidden>
Date: Fri Nov 22 13:38:47 2013 -0600

    recommend excluding 35357 from ephemeral ports

    Change-Id: I56c3b526ac4af255dd31ce0bd0e0f2ce128cb6fe
    Closes-Bug: 1253482

The e-r query on this bug seems to not be working anymore, it shows no hits in 14 days but I hit it here:

http://logs.openstack.org/93/98493/1/check/check-swift-dsvm-functional/ffb861b/logs/screen-key.txt.gz#_2014-06-09_14_08_14_587

Fix proposed to branch: master
Review: https://review.openstack.org/99779

Reviewed: https://review.openstack.org/99779
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=6cae83efd72133adae891da0665f51f526705592
Submitter: Jenkins
Branch: master

commit 6cae83efd72133adae891da0665f51f526705592
Author: Morgan Fainberg <email address hidden>
Date: Thu Jun 12 15:08:48 2014 -0700

    Reserve Keystone ports from the ephemeral range

    Reserve Keystone ports from the ephemeral range as early as reasonably
    possible in the fixup_stuff.sh process to reduce the likelihood that the
    port will be in use. This does not completely resolve the issue
    where Keystone's IANA assigned port falls into Linux's ephemeral
    range, but this should reduce the occurrences. The default ports
    are 35357 and 35358.

    Change-Id: I8cfb53d8f90c1ff1fb1083c59fefabca3d14323b
    Partial-Bug: #1253482

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/130187

Reviewed: https://review.openstack.org/130187
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=80cad72055db14e6bc7c610c470f9977266e1c65
Submitter: Jenkins
Branch: stable/icehouse

commit 80cad72055db14e6bc7c610c470f9977266e1c65
Author: Morgan Fainberg <email address hidden>
Date: Thu Jun 12 15:08:48 2014 -0700

    Reserve Keystone ports from the ephemeral range

    Reserve Keystone ports from the ephemeral range as early as reasonably
    possible in the fixup_stuff.sh process to reduce the likelihood that the
    port will be in use. This does not completely resolve the issue
    where Keystone's IANA assigned port falls into Linux's ephemeral
    range, but this should reduce the occurrences. The default ports
    are 35357 and 35358.

    Change-Id: I8cfb53d8f90c1ff1fb1083c59fefabca3d14323b
    Partial-Bug: #1253482
    (cherry picked from commit 6cae83efd72133adae891da0665f51f526705592)