[RFE] Push all object information in AMQP notifications
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| neutron |
Won't Fix
|
Wishlist
|
Kevin Benton | ||
Bug Description
The agent/server communication pattern we use now can lead to cascading failures making the servers unavailable.
The current pattern in our communications between the Neutron server and the agents looks like the following:
Server sends: item <item-uuid> changed
Client receives event.
Client makes a call to the server asking for the item details.
The calls the client makes to the server can be expensive and a server under heavy load can take a long time to start processing the request and/or to fulfill the request. This can trigger a timeout on the agent side, which leads to a retry, or, even worse, a generic fallback to resync the entire state. This creates a thundering herd problem where a server falling behind on requests will be continually stampeded by retries from agents that have timed out by the time the server can respond.
The pattern of agent/server communication needs to be adjusted to assume terrible server response times at a minimum. Optimally, all of the notifications generated by the servers should be adjusted to include all of the relevant information that an agent will need to respond to an event so the only time an agent has to actually call the server is on startup to get initial state.
| Miguel Angel Ajo (mangelajo) wrote | #1 |
| Changed in neutron: | |
| importance: | Undecided → Medium |
| tags: | added: loadimpact |
| Changed in neutron: | |
| status: | New → Triaged |
| Armando Migliaccio (armando-migliaccio) wrote | #2 |
| tags: | added: rfe |
| Changed in neutron: | |
| importance: | Medium → Wishlist |
| Armando Migliaccio (armando-migliaccio) wrote | #3 |
| tags: |
added: rfe-approved removed: rfe |
| Changed in neutron: | |
| assignee: | nobody → Kevin Benton (kevinbenton) |
| milestone: | none → mitaka-2 |
| Changed in neutron: | |
| milestone: | mitaka-2 → mitaka-3 |
| summary: |
- Push all object information in AMQP notifications + [RFE] Push all object information in AMQP notifications |
| Changed in neutron: | |
| milestone: | mitaka-3 → mitaka-rc1 |
| Armando Migliaccio (armando-migliaccio) wrote | #4 |
| Changed in neutron: | |
| milestone: | mitaka-rc1 → newton-1 |
| OpenStack Infra (hudson-openstack) wrote Related fix merged to neutron-specs (master) | #5 |
| Changed in neutron: | |
| milestone: | newton-1 → newton-2 |
| Changed in neutron: | |
| milestone: | newton-2 → newton-3 |
| Changed in neutron: | |
| milestone: | newton-3 → newton-rc1 |
| Changed in neutron: | |
| milestone: | newton-rc1 → ocata-1 |
| Changed in neutron: | |
| milestone: | ocata-1 → ocata-2 |
| Changed in neutron: | |
| milestone: | ocata-2 → ocata-3 |
| Changed in neutron: | |
| milestone: | ocata-3 → ocata-rc1 |
| Changed in neutron: | |
| milestone: | ocata-rc1 → ongoing |
| Changed in neutron: | |
| milestone: | ongoing → pike-1 |
| Changed in neutron: | |
| milestone: | pike-1 → pike-2 |
| Brian Haley (brian-haley) wrote | #6 |
| Changed in neutron: | |
| status: | Triaged → Won't Fix |
Yes, we need a good RPC mechanism to avoid the need to sync-back on neutron server.
And probably, implementing step back / circuit breaking patterns on requests back to neutron to mitigate cascading failures.
If we used a single source of incremental IDs we could rely on (redis INCR [1]? or any abstracted client..), we could tag RPC messages with incremental IDs to avoid the out-of-order issues. Or, we could timestamp resources on DB to make sure we always keep the latest update of an object (but that gets complicated on composited objects: like when we add "qos_policy_id" by extending a port)..