Fuel for OpenStack

Task serialization when YAQL used takes too much time

Bug #1572103 reported by Stanislaw Bogatkin
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
In Progress
High
Bulat Gaifullin
Fuel for OpenStack 10.1
Mitaka
Fix Released
High
Bulat Gaifullin
Fuel for OpenStack 9.0

Bug Description

Fuel 10.0, current master

When we use tasks with YAQL conditions on environment with many nodes (200+), it takes really long time to get serialization done.
On machine with Intel(R) Xeon(R) CPU E5-2620 (12 cores) we have next serialization time:

5 nodes:
2016-04-15 14:15:49.834 tasks serialization is started.
2016-04-15 14:16:05.277 tasks serialization is finished.
diff = 16s

50 nodes:
2016-04-15 14:30:29.376 tasks serialization is started.
2016-04-15 14:33:25.227 tasks serialization is finished.
diff: 176s (2:56)

200 nodes:
VSIZE: 1.2G, RSIZE: 488.3M
2016-04-15 14:39:25.297 tasks serialization is started.
2016-04-15 15:09:05.396 tasks serialization is finished.
diff: 1780s (29:40)

500 nodes:
VSIZE:2.2G, RSIZE: 1.6G
2016-04-15 15:33:56.381 tasks serialization is started.
2016-04-15 18:08:02.706 tasks serialization is finished
diff: 9246s (2:34:06)

It's really much, we need to find a way to improve this.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/307740

Changed in fuel:
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Bulat Gaifullin (bgaifullin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/318206

Changed in fuel:
assignee: Bulat Gaifullin (bgaifullin) → Stanislaw Bogatkin (sbogatkin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/318206
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=9a806f1d12ed45d7fa623dfb0dec664a8bcb2328
Submitter: Jenkins
Branch: master

commit 9a806f1d12ed45d7fa623dfb0dec664a8bcb2328
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed May 18 19:28:36 2016 +0300

    Use C vairant of YAML dumper at tasks serialization

    To speed up seiralization time, use faster implementation of
    YAML dumping when it is possible.

    Change-Id: I992410328c07d14a1f5903aa91d456b040175a69
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Bulat Gaifullin (bgaifullin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/322217

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/322217
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=6dea7664c2365bc5fd6ac72e80943d3136a5880f
Submitter: Jenkins
Branch: stable/mitaka

commit 6dea7664c2365bc5fd6ac72e80943d3136a5880f
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed May 18 19:28:36 2016 +0300

    Use C vairant of YAML dumper at tasks serialization

    To speed up seiralization time, use faster implementation of
    YAML dumping when it is possible.

    Change-Id: I992410328c07d14a1f5903aa91d456b040175a69
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>
    (cherry picked from commit 9a806f1d12ed45d7fa623dfb0dec664a8bcb2328)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/322960

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/307740
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=02c4283fe7ac850c0d51e03ee9c05cc2ee3a2f60
Submitter: Jenkins
Branch: master

commit 02c4283fe7ac850c0d51e03ee9c05cc2ee3a2f60
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Apr 19 14:55:57 2016 +0300

    Multiprocess tasks serialization

    Tasks serialization process take long time in case when environment
    contains many nodes. To reduce time of serialization, make
    serialization process work in parallel (reduces time of
    serialization mostly linear way with increasing workers pool)

    DocImpact
    Change-Id: Id3753dbc6983256d410e69c98ab02b61ab6bfb7f
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>

Revision history for this message
Bulat Gaifullin (bulat.gaifullin) wrote :

ETA: 1 day

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/322960
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=4efe81f5894299f361fede9836c65f4ec4ba7853
Submitter: Jenkins
Branch: stable/mitaka

commit 4efe81f5894299f361fede9836c65f4ec4ba7853
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue Apr 19 14:55:57 2016 +0300

    Multiprocess tasks serialization

    Tasks serialization process take long time in case when environment
    contains many nodes. To reduce time of serialization, make
    serialization process work in parallel (reduces time of
    serialization mostly linear way with increasing workers pool)

    DocImpact
    Change-Id: Id3753dbc6983256d410e69c98ab02b61ab6bfb7f
    Partial-Bug: #1572103
    Co-Authored-With: V. Kuklin <email address hidden>
    Co-Authored-With: B. Gaifullin <email address hidden>
    (cherry picked from commit 02c4283fe7ac850c0d51e03ee9c05cc2ee3a2f60)

Maksim Malchuk (mmalchuk)
no longer affects: fuel/newton
Kyrylo Galanov (kgalanov)
tags: added: on-verification
Revision history for this message
Kyrylo Galanov (kgalanov) wrote :

Verified on ISO #465 RC1

tags: removed: on-verification
Dmitry Pyzhov (dpyzhov)
tags: added: 9.1-proposed
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 10.0 → 10.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.