Unequal distribution across OSDs

Bug #1402080 reported by Vinod Nair
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Won't Fix
High
Jeya ganesh babu J
R2.0
Won't Fix
High
Jeya ganesh babu J
R2.1
Won't Fix
High
Jeya ganesh babu J
R2.20
Won't Fix
High
Jeya ganesh babu J
R3.0
Won't Fix
Undecided
Unassigned

Bug Description

Ceph , on writing objects some of the disks are getting full very much ahead. looks like the crush map calculation is not correct

osdstat kbused kbavail kb hb in hb out
0 792044472 183702072 975746544 [1,3,4,5,6,7,8,9,10,11,15] []
1 790672896 185073648 975746544 [0,2,3,4,5,6,7,8,9,10,11] []
2 700233716 275512828 975746544 [1,3,4,5,6,7,8,9,10,11] []
3 927252696 48493848 975746544 [0,1,2,4,6,7,8,9,10,11] [] ><<<<<<<<< This is nearly full
4 654742612 321003932 975746544 [0,1,2,3,5,6,7,8,9,10,11] []
5 568450712 407295832 975746544 [0,1,2,4,6,7,8,9,10,11] []
6 793254988 182491556 975746544 [0,1,2,3,4,5,7,9,10,11] []
7 816994332 158752212 975746544 [0,1,2,3,4,5,6,8,9,10,11] []
8 724516484 251230060 975746544 [0,1,2,3,4,5,7,9,10,11] []
9 755386980 220359564 975746544 [0,1,2,3,4,5,6,7,8,10] []
10 758759272 216987272 975746544 [0,1,2,3,4,5,6,7,8,9,11] []
11 636963552 338782992 975746544 [0,1,2,3,4,5,6,7,8,10,12] []
12 1149376 387779584 388928960 [0,1,6,8,9,10,11,13,14,15] []
13 1243804 387685156 388928960 [4,5,6,7,8,9,10,12,14,15] []
14 1272632 387656328 388928960 [0,1,4,6,9,10,11,12,13,15] []
15 1358920 387570040 388928960 [0,4,5,6,7,8,11,12,13,14] []
 sum 8924297444 4340376924 13264674368

version: 14.04 Ubuntu Icehouse , Similar behaviour is seen in 12.04 also : build : 2:13

Revision history for this message
Michael Nishimoto (mnishimoto) wrote :

This will be a problem for any serious setup, so we need to fix it for 2.0.

tags: added: blocker
summary: - Unequal distribution accross OSD's
+ Unequal distribution across OSD's
tags: removed: blocker
tags: added: releasenote
kamlesh parmar (kparmar)
information type: Proprietary → Public
Revision history for this message
Michael Nishimoto (mnishimoto) wrote : Re: Unequal distribution across OSD's

Suggestion from CERN for their 3PB install is to use: reweight-by-utilization feature of Ceph to fix this issue.

summary: - Unequal distribution across OSD's
+ Unequal distribution across OSDs
Revision history for this message
Jeya ganesh babu J (jjeya) wrote :

Release note: User may see unequal usage of the disks under Ceph. To work around, use osd reweight to balance out the disk manually.

tags: removed: releasenote
tags: added: releasenote
Changed in juniperopenstack:
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.