incompatible pickle protocol between python versions in Listener.insert_headers

Originally reported by Tobias Urdin at https://storyboard.openstack.org/#!/story/2004255

The type of the insert_headers column in the listener DB table is not always compatible across python version.
When doing rolling upgrades for instance, a controller with python 3.9 may write an entry in the database that cannot be read by older controllers (python 3.6)

2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server [-] Exception during message handling: ValueError: unsupported pickle protocol: 5
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/controller/queue/v2/endpoints.py", line 115, in create_member
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server self.worker.create_member(member)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 333, in wrapped_f
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return self(f, *args, **kw)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 423, in __call__
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 360, in iter
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return fut.result()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return self.__get_result()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server raise self._exception
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 426, in __call__
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v2/controller_worker.py", line 419, in create_member
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server id=member[constants.MEMBER_ID])
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/db/repositories.py", line 138, in get
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return model.to_data_model()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/db/base_models.py", line 94, in to_data_model
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server _graph_nodes=_graph_nodes))
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/db/base_models.py", line 94, in to_data_model
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server _graph_nodes=_graph_nodes))
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/octavia/db/base_models.py", line 85, in to_data_model
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server attr = getattr(self, attr_name)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 481, in __get__
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return self.impl.get(state, dict_)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 941, in get
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server value = self._fire_loader_callables(state, key, passive)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 977, in _fire_loader_callables
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return self.callable_(state, passive)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 917, in _load_for_state
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server extra_criteria,
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 1051, in _emit_lazyload
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server result = result.unique().scalars().all()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 1361, in all
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return self._allrows()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 400, in _allrows
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server rows = self._fetchall_impl()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 1274, in _fetchall_impl
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return self._real_result._fetchall_impl()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 1686, in _fetchall_impl
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return list(self.iterator)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server fetch = cursor._raw_all_rows()
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 392, in _raw_all_rows
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return [make_row(row) for row in rows]
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/result.py", line 392, in <listcomp>
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return [make_row(row) for row in rows]
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/site-packages/sqlalchemy/sql/sqltypes.py", line 1868, in process
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server return loads(value)
2023-09-13 22:16:06.961 3126384 ERROR oslo_messaging.rpc.server ValueError: unsupported pickle protocol: 5

The type of the data is PickleType (defined by SQLAlchemy)

https://opendev.org/openstack/octavia/src/commit/d7bba6c5af3c0f6d99ea8063b54ec3e895a5eea4/octavia/db/models.py#L591

https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.PickleType

By default it uses pickle.HIGH_PROTOCOL which may change depending on the python version

https://docs.python.org/3/library/pickle.html#data-stream-format

For instance, in python 3.9, the data will be encoded with protocol version 5, which is not readable by python 3.6

To solve this issue, we could force the use of protocol version 4, this version is supported by all the python versions (>= python 3.4) used by the stable releases of OpenStack.

As a next step, we could also change the type to json, but it would introduce some changes in the DB (need to convert existing data) and that would be a non-backportable patch.