We have looked a bit deeper both into the test itself but also looking for how other "people" manage this. I'm not going to echo much as I think that this comment (https://bugs.linaro.org/show_bug.cgi?id=3402#c9) from Linaro nicely describes both the problem and how this is an unreasonably unstable test. Upstream was contacted for a tolerance increase in the test but no feedback was received (https://<email address hidden>/).
We have been dragging this test for a good while blocked in a hinting-loop to ignore the results of this test which is costing us time to no benefits at all. I propose and will go ahead with patch ml submissions to avoid failing the test even if the 10% threshold is passed. I propose to go with this as opposed to a full revert of 0b63accf87225b5eb7e52814c374cf02d733d4bb for the following reasons:
1. it keeps running the test with results and data that we can use in the future for any "smarter" way to deal with it - eg. statistically
2. it minimizes the maintenance burden of that patch kept in our kernel trees (a simple exit line removal)
We have looked a bit deeper both into the test itself but also looking for how other "people" manage this. I'm not going to echo much as I think that this comment (https:/ /bugs.linaro. org/show_ bug.cgi? id=3402# c9) from Linaro nicely describes both the problem and how this is an unreasonably unstable test. Upstream was contacted for a tolerance increase in the test but no feedback was received (https://<email address hidden>/).
We have been dragging this test for a good while blocked in a hinting-loop to ignore the results of this test which is costing us time to no benefits at all. I propose and will go ahead with patch ml submissions to avoid failing the test even if the 10% threshold is passed. I propose to go with this as opposed to a full revert of 0b63accf87225b5 eb7e52814c374cf 02d733d4bb for the following reasons:
1. it keeps running the test with results and data that we can use in the future for any "smarter" way to deal with it - eg. statistically
2. it minimizes the maintenance burden of that patch kept in our kernel trees (a simple exit line removal)