OpenStack Identity (keystone)

"Too many connections" with MySQL + sql token driver + v3 auth

Bug #1162845 reported by Dolph Mathews
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
High
Dolph Mathews
OpenStack Identity (keystone) 2013.1 "grizzly"

Bug Description

Reported on the mailing list by Miller, Mark M:

  https://lists.launchpad.net/openstack/msg22307.html

Excerpt:

  (keystone.auth.controllers): ... ERROR (OperationalError) (1040, 'Too many connections') None None
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py", line 286, in authenticate_for_token
      self.authenticate(context, auth_info, auth_context)
  ...
    File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
      super(Connection, self).__init__(*args, **kwargs2)
  OperationalError: (OperationalError) (1040, 'Too many connections') None None
  (access): ... INFO ... "POST http://15.253.58.165:35357/v3/auth/tokens HTTP/1.0" 401 125
  (eventlet.wsgi.server): ... DEBUG ... "POST /v3/auth/tokens HTTP/1.1" 401 290 0.017322

Revision history for this message
Dolph Mathews (dolph) wrote :

Attempted to reproduce this with PKI + sqlite against v2 does not appear to reproduce the issue. However, switching to v3 (as indicated by the above backtrace) revealed a slightly different symptom, documented in bug 1162857. These two bugs may share the same underlying cause.

Thierry Carrez (ttx)
tags: added: grizzly-rc-potential
Changed in keystone:
importance: Undecided → High
Dolph Mathews (dolph)
summary: - "Too many connections" with PKI + MySQL
+ "Too many connections" with MySQL
Revision history for this message
Dolph Mathews (dolph) wrote : Re: "Too many connections" with MySQL

I was able to reproduce this with both UUID and PKI tokens against a mysql:// sqlalchemy connection string and explicitly setting the token driver to the sql-based one (which is not the default). This bug appears to affect v3 auth, but not v2 auth.

Changed in keystone:
status: New → Confirmed
summary: - "Too many connections" with MySQL
+ "Too many connections" with MySQL + sql token driver + v3 auth
OpenStack Infra (hudson-openstack)
Changed in keystone:
assignee: nobody → Dolph Mathews (dolph)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25833

Thierry Carrez (ttx)
no longer affects: keystone/grizzly
Changed in keystone:
milestone: none → grizzly-rc3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/25831
Committed: http://github.com/openstack/keystone/commit/a2e3a9228afe5bc0a42bb168b728caf29e394b3d
Submitter: Jenkins
Branch: master

commit a2e3a9228afe5bc0a42bb168b728caf29e394b3d
Author: Dolph Mathews <email address hidden>
Date: Mon Apr 1 14:41:26 2013 -0500

    Share one engine for more than just sqlite in-memory

    This fixes both bug 1162857 and bug 1162845.

    Change-Id: Ica5155d3ac03779693d2d54d8e02c2a413c3d3ac

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (milestone-proposed)

Reviewed: https://review.openstack.org/25833
Committed: http://github.com/openstack/keystone/commit/93b8f2ddf7f00747d7ccace8401b2b68a11bf98f
Submitter: Jenkins
Branch: milestone-proposed

commit 93b8f2ddf7f00747d7ccace8401b2b68a11bf98f
Author: Dolph Mathews <email address hidden>
Date: Mon Apr 1 14:41:26 2013 -0500

    Share one engine for more than just sqlite in-memory

    This fixes both bug 1162857 and bug 1162845.

    Change-Id: Ica5155d3ac03779693d2d54d8e02c2a413c3d3ac

Changed in keystone:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in keystone:
milestone: grizzly-rc3 → 2013.1
Revision history for this message
Thiago Martins (martinx) wrote :

Guys,

 I am seeing this problem on Kilo.

 What can I do? Re-open it? Or fill another bug report?

Thanks,
Thiago

Revision history for this message
Dolph Mathews (dolph) wrote :

Because this bug report has long been closed, it'd be best to open a new bug and reference this one as a regression.

That said, what's your SQL connection limit? We've hit kilo pretty hard and I haven't seen this sort of misbehavior, although it's possible to scale your Keystone cluster beyond SQL's default limits.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.