OpenStack Compute (nova)

Qpid cannot serialize objects containing strings longer than 65535 characters

Bug #1175808 reported by Ben Nemec on 2013-05-02

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
OpenStack Compute (nova)	Fix Released	Critical	Andrew Laski	OpenStack Compute (nova) 2013.2 "havana"
Grizzly	Fix Released	Critical	Ben Nemec	OpenStack Compute (nova) 2013.1.3
OpenStack Heat	Fix Released	High	GuoHui Liu	OpenStack Heat 2013.2 "havana"
neutron	Fix Released	Critical	Ben Nemec	neutron 2013.2 "havana"
Grizzly	Fix Released	High	Ben Nemec	neutron 2013.1.3
oslo-incubator	Fix Released	Critical	Ben Nemec	oslo-incubator 2013.2 "havana"

Bug Description

Qpid has a limitation where it cannot serialize a Python dict containing a string longer than 65535 characters. This can result in problems when making a conductor call that returns a large structure - for example, instance_get_all_by_host on one of my systems returns 38 instances, which when serialized as JSON is too long for Qpid to handle.

There's a discussion of the limitation here: https://issues.apache.org/jira/browse/QPID-4583

I believe the best fix here is to wrap the string in a buffer as suggested in the link above. This works fine on my local system and seems to fix the problem (note: I haven't tested against RabbitMQ yet, and right now I put the change in common code so that will need to be done). However, that will be a backward compatibility breaking change because it will need to be handled differently on the receiving end as well.

Revision history for this message

Ben Nemec (bnemec) wrote on 2013-05-02:

Stack trace when trying to send a long string through qpid Edit (5.4 KiB, text/plain)

Revision history for this message

Michael Still (mikal) wrote on 2013-05-08:

This is implemented in oslo.

Changed in nova:
status:	New → Invalid

Ben Nemec (bnemec) on 2013-05-09

Changed in oslo:
assignee:	nobody → Ben Nemec (bnemec)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-05-09: Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/28711

Changed in oslo:
status:	New → In Progress

Revision history for this message

Ben Nemec (bnemec) wrote on 2013-05-09:

One note on my proposed fix: I ended up not wrapping the long string in a buffer because that required the qpid code to know about the message format, which seemed like a bad thing. This change is transparent to external code and the only incompatibility is that the old implementation won't be able to handle the new messages, but the other direction will work fine.

Mark McLoughlin (markmc) on 2013-05-13

tags:	added: grizzly-backport-potential
Changed in oslo:
importance:	Undecided → Critical

Revision history for this message

Mark McLoughlin (markmc) wrote on 2013-05-13:

The code is in oslo-incubator but does need to be synced across to Nova, so think we should keep a Nova task open

Changed in nova:
status:	Invalid → Triaged
importance:	Undecided → Critical

Revision history for this message

Steven Hardy (shardy) wrote on 2013-05-14:

Added Heat, as you can see from the qpid JIRA I've already been down the path of investigating this, and came to the conclusion (for the Heat use-case at least) that the best option was *not* to JSON encode, since the recursive map serialization works OK for the data we typically need to pass (where none of individual the dict/map values contain strings >65535 but the JSON encoded entire template can easily be)

If there is a transparent workaround in oslo then we should sync that into Heat, as it's still possible users could encounter this limitation if they put particularly huge strings into a Heat template.

bug #1072917 has further details on my previous investigation into this.

Revision history for this message

Russell Bryant (russellb) wrote on 2013-05-22:

Doing encoding before it hits qpid is required so that we can do message signing, though.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-04: Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/28711
Committed: http://github.com/openstack/oslo-incubator/commit/7ce54410485b33cffc39c7ffb96eae422b88601c
Submitter: Jenkins
Branch: master

commit 7ce54410485b33cffc39c7ffb96eae422b88601c
Author: Ben Nemec <email address hidden>
Date: Thu May 9 19:06:45 2013 +0000

Fix problem with long messages in Qpid

    Qpid has a limitation where it cannot serialize a dict containing
    a string greater than 65535 characters. This change alters the
    Qpid implementation to JSON encode the dict before sending it, but
    only if Qpid would fail to serialize it. This maintains as much
    backward compatibility as possible, though long messages will
    still fail if they are sent to an older receiver.

Fixes bug 1175808

Change-Id: I5d104e099f523508dae2b657f7d06d96984b10f0

Changed in oslo:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-04: Fix proposed to oslo-incubator (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/31689

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-05: Fix proposed to nova (master)

#10

Fix proposed to branch: master
Review: https://review.openstack.org/31803

Changed in nova:
assignee:	nobody → Andrew Laski (alaski)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-05: Fix merged to nova (master)

#11

Reviewed: https://review.openstack.org/31803
Committed: http://github.com/openstack/nova/commit/781a8f908cd3e5e69ff8b88d998fa93c48532e15
Submitter: Jenkins
Branch: master

commit 781a8f908cd3e5e69ff8b88d998fa93c48532e15
Author: Andrew Laski <email address hidden>
Date: Wed Jun 5 10:02:07 2013 -0400

Update rpc/impl_qpid.py from oslo

    The current qpid driver cannot serialize objects containing strings
    longer than 65535 characters. This just became a breaking issue when
    the message to scheduler_run_instance went over that limit. The fix has
    been commited to oslo, so this just syncs it over to Nova.

Bug 1175808
Bug 1187595

Change-Id: If95c11a7e03c81d89133f6cad0dcbb6d8acb8148

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-07: Fix proposed to quantum (master)

#12

Fix proposed to branch: master
Review: https://review.openstack.org/32173

Changed in quantum:
assignee:	nobody → Ben Nemec (bnemec)
status:	New → In Progress

Mark McClain (markmcclain) on 2013-06-07

Changed in quantum:
importance:	Undecided → Critical

GuoHui Liu (guohliu) on 2013-06-08

Changed in heat:
assignee:	nobody → GuoHui LIu (guohliu)

GuoHui Liu (guohliu) on 2013-06-08

Changed in heat:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-09: Fix proposed to heat (master)

#13

Fix proposed to branch: master
Review: https://review.openstack.org/32291

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-09: Fix merged to quantum (master)

#14

Reviewed: https://review.openstack.org/32173
Committed: http://github.com/openstack/quantum/commit/68a5a387020cdb0c5d664b7add2d897134877d19
Submitter: Jenkins
Branch: master

commit 68a5a387020cdb0c5d664b7add2d897134877d19
Author: Ben Nemec <email address hidden>
Date: Fri Jun 7 16:35:14 2013 +0000

Sync Qpid RPC fix from Oslo

Qpid cannot serialize dicts containing strings longer than 65535
characters. This change syncs the fix from Oslo to Quantum.

Fixes bug 1175808

Change-Id: I48071abffa86e71727deed05aca08ac475cbaf05

Changed in quantum:
status:	In Progress → Fix Committed

Mark McClain (markmcclain) on 2013-06-10

Changed in quantum:
milestone:	none → havana-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-18: Fix merged to oslo-incubator (stable/grizzly)

#15

Reviewed: https://review.openstack.org/31689
Committed: http://github.com/openstack/oslo-incubator/commit/860827fafcc020ab0d7f9d56b6f03b6e8b8ebc7b
Submitter: Jenkins
Branch: stable/grizzly

commit 860827fafcc020ab0d7f9d56b6f03b6e8b8ebc7b
Author: Ben Nemec <email address hidden>
Date: Tue Jun 4 18:52:06 2013 +0000

Grizzly compatibility with Havana Qpid messages

    A change had to be made in the Havana Qpid implementation that is
    not entirely backwards compatible. This change allows Grizzly to
    receive messages in the new format.

Backport of change for bug 1175808

Change-Id: I8b6c5734b67cd33cdd5ef0659bdd77504317187d

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-18: Fix proposed to nova (stable/grizzly)

#16

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/33476

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-18: Fix proposed to quantum (stable/grizzly)

#17

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/33478

Steven Hardy (shardy) on 2013-06-20

Changed in heat:
importance:	Undecided → High
milestone:	none → havana-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-20: Fix merged to heat (master)

#18

Reviewed: https://review.openstack.org/32291
Committed: http://github.com/openstack/heat/commit/fee94d28e780eb63be2bed608ecaab5750c07267
Submitter: Jenkins
Branch: master

commit fee94d28e780eb63be2bed608ecaab5750c07267
Author: guohliu <email address hidden>
Date: Wed Jun 19 20:18:39 2013 +0800

Updated common module from oslo

This patch updated common module from oslo for known issue.

Fixed bug: #1175808
Fixed bug: #1072917

Change-Id: Ic1de6b786a9d67b39d186e31a37049729d367464

Changed in heat:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-06-20: Fix merged to quantum (stable/grizzly)

#19

Reviewed: https://review.openstack.org/33478
Committed: http://github.com/openstack/quantum/commit/a99cc45585759014c14491ab5eb45429f207e749
Submitter: Jenkins
Branch: stable/grizzly

commit a99cc45585759014c14491ab5eb45429f207e749
Author: Ben Nemec <email address hidden>
Date: Tue Jun 18 17:11:33 2013 +0000

Sync Qpid RPC fix from Oslo

    A change had to be made in the Havana Qpid implementation that is
    not entirely backwards compatible. This change allows Grizzly to
    receive messages in the new format.

Backport of change for bug 1175808

Change-Id: I6ab63c3fccf562aeeb793e09f057c66bc109ef41

Gary Kotton (garyk) on 2013-06-24

tags:

added: in-stable-grizzly
removed: grizzly-backport-potential

Alan Pevec (apevec) on 2013-07-09

tags:

removed: in-stable-grizzly

Alan Pevec (apevec) on 2013-07-09

Changed in nova:
milestone:	none → havana-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-07-09: Fix merged to nova (stable/grizzly)

#20

Reviewed: https://review.openstack.org/33476
Committed: http://github.com/openstack/nova/commit/d2cb4b25f7226cd2051d38905942e515d257931c
Submitter: Jenkins
Branch: stable/grizzly

commit d2cb4b25f7226cd2051d38905942e515d257931c
Author: Ben Nemec <email address hidden>
Date: Tue Jun 18 16:59:04 2013 +0000

Sync RPC module from Oslo

Includes backport of change for bug 1175808
Fixes bug 1172922 and bug 1122763

Change-Id: Ib52e9458a9db01e2c8fd2a6d84010733f85980f8

Thierry Carrez (ttx) on 2013-07-17

Changed in heat:
status:	Fix Committed → Fix Released
Changed in nova:
status:	Fix Committed → Fix Released
Changed in neutron:
status:	Fix Committed → Fix Released
Changed in oslo:
milestone:	none → havana-2
status:	Fix Committed → Fix Released