Neutron failed to create network

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

Can you please include full traceback to the description? Currently it doesn't even show what the Error is. I see snapshot attached but full tracebacks would really help triaging bugs

2016-06-22 15:00:15.516 23028 ERROR neutron.api.v2.resource OperationalError: (_mysql_exceptions.OperationalError) (1305, 'SAVEPOINT sa_savepoint_1 does not exist') [SQL: u'ROLLBACK TO SAVEPOINT sa_savepoint_1']

On security group creation during network creation.
And here we go again

This script reproduce similar exception which Eugene mentioned above:

http://paste.openstack.org/show/562439/

Looks like a bug in sqlalchemy:

https://bitbucket.org/zzzeek/sqlalchemy/issues/2696/misleading-exception-triggered-on

Traceback generated by script above:

(myenv) ➜ myenv python new_script.py
2016-08-23 13:41:26,665 INFO sqlalchemy.engine.base.Engine SHOW VARIABLES LIKE 'sql_mode'
2016-08-23 13:41:26,665 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,668 INFO sqlalchemy.engine.base.Engine SELECT DATABASE()
2016-08-23 13:41:26,668 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,671 INFO sqlalchemy.engine.base.Engine show collation where `Charset` = 'utf8' and `Collation` = 'utf8_bin'
2016-08-23 13:41:26,671 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,673 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS CHAR(60)) AS anon_1
2016-08-23 13:41:26,673 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,675 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS CHAR(60)) AS anon_1
2016-08-23 13:41:26,675 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,676 INFO sqlalchemy.engine.base.Engine SELECT CAST('test collated returns' AS CHAR CHARACTER SET utf8) COLLATE utf8_bin AS anon_1
2016-08-23 13:41:26,676 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,677 INFO sqlalchemy.engine.base.Engine DESCRIBE `a`
2016-08-23 13:41:26,677 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,685 INFO sqlalchemy.engine.base.Engine ROLLBACK
2016-08-23 13:41:26,686 INFO sqlalchemy.engine.base.Engine DESCRIBE `a`
2016-08-23 13:41:26,686 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,687 INFO sqlalchemy.engine.base.Engine ROLLBACK
2016-08-23 13:41:26,688 INFO sqlalchemy.engine.base.Engine
CREATE TABLE a (
        id INTEGER NOT NULL,
        PRIMARY KEY (id)
)

2016-08-23 13:41:26,688 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,741 INFO sqlalchemy.engine.base.Engine COMMIT
2016-08-23 13:41:26,742 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2016-08-23 13:41:26,742 INFO sqlalchemy.engine.base.Engine SAVEPOINT sa_savepoint_1
2016-08-23 13:41:26,742 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,743 INFO sqlalchemy.engine.base.Engine rollback
2016-08-23 13:41:26,743 INFO sqlalchemy.engine.base.Engine {}
2016-08-23 13:41:26,745 INFO sqlalchemy.engine.base.Engine INSERT INTO a () VALUES ()
2016-08-23 13:41:26,745 INFO sqlalchemy.engine.base.Engine {}
/Users/aignatov/work/myenv/lib/python2.7/site-packages/pymysql/cursors.py:167: Warning: (1364, "Field 'id' doesn't have a default value")
  result = self._query(query)
2016-08-23 13:41:26,752 INFO sqlalchemy.engine.base.Engine ROLLBACK TO SAVEPOINT sa_savepoint_1
2016-08-23 13:41:26,752 INFO sqlalchemy.engine.base.Engine {}
/Users/aignatov/work/myenv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:430: SAWarning: An exception raised during a Session persistence operation cannot be raised due to an additional ROLLBACK exception; the exception is: Instance <A at 0x104a5e650> has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such aswithin a load() event.
...

Full exception in neutron-server-1 from node-1, taken from snapshot:
http://paste.openstack.org/show/562448/

Traceback is similar to what we have in comment #5.

The only way here is to update sqlalchemy to a higher version to catch original exception which is currently being interrupted by exception during attempt to do rollback. To be more precise, this fix will help with further investigations which is in sqlalchemy==1.0.12:

https://bitbucket.org/zzzeek/sqlalchemy/commits/8a1e619fb20df1be6ad2e0c563e451e17eb17628#chg-lib/sqlalchemy/util/langhelpers.py

Hmm, now I've realized that in mos 9.0 we already use sqlalchemy==1.0.12 but still it's not clear why we don't see this specified WARNING in neutron logs:

https://bitbucket.org/zzzeek/sqlalchemy/commits/8a1e619fb20df1be6ad2e0c563e451e17eb17628#chg-lib/sqlalchemy/util/langhelpers.py

Looks like Kevin's fix should help with SAVEPOINT error.

https://review.openstack.org/#/c/326927/

Looks like we'll be able to fix this by adding pymsql driver first than backport Kevin's patch.

Since we are not going to put "pymysql" driver as defualt driver in mos, moving this issue to 9.2.

No plans to fix it in 9.2 by moving MOS to pymysql driver, targeting this bug to 10.0