cinder-volume disposes mysql pool every time it creates a volume

Bug #1773764 reported by cheng li
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Confirmed
Medium
Unassigned

Bug Description

I create volumes with concurrency of 500, so I found huge number of TCP connections to mysql in *time_wait* status(on the server running cinder-volume). After going through the cinder-volume code, I found that every time I create a volume the connection pool is disposed. That's why so many *time_wait*.
Seems this bug was introduced in this [commit](https://review.openstack.org/#/c/163551/4/cinder/db/base.py). Michael and the patch author Eric they discussed about the potential issue.

"However, engine.dispose() shouldn't really be called anywhere else under normal circumstances because it closes out the whole connection pool, so this Base object needs to be something that is not generally created multiple times in-process."

They assumed that engine.dispose() will not be called multiple times in-process, but now it is. Every time we create a volume, engine.dispose() is called. Because of this line. https://github.com/openstack/cinder/blob/970b8cb94ce6a0d6024305fc949060c5d1b589ee/cinder/volume/flows/manager/create_volume.py#L379

I was going through the code of Pike version, the I checked master branch, seems master branch also has this issue.

Seems we can't assume engine.dispose() will only be called one time in process.Because there are many authors commit code. I am wondering if we can add a parameter(let's name it "dispose_db_engine" of default value False) for base.Base class https://github.com/openstack/cinder/blob/5dcf4f52ada36b98d08584cd31a55698006d9e35/cinder/db/base.py#L36

Tags: dispose mysql
Changed in cinder:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Gorka Eguileor (gorka) wrote :

According to your analysis, wouldn't we solve the issue if we just replaced the `message` instance attribute with a class attribute?

Revision history for this message
cheng li (chenga) wrote :

I think that what Eric thought -- call dispose_engine() only one time. But there are many people contributing to cinder project, it's hard to ensure every one of us will keep in mind that not creating new message API instance. I think that why this issue happens, people didn't follow Eric's rule.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.