kubuntu-settings-desktop breaks subpixel font rendering on XFCE

I am a Xubuntu/XFCE user and noticed that my subpixel rendering was broken. Even so the subpixel order was set to "None", fonts would render with subpixel order "RGB". If I set the subpixel order to BGR, Vertical RGB/BGR however things would work. Only the 'None' option had the problem.

The problem turned out to be this symlink from the kubuntu-settings-desktop package:

/etc/fonts/conf.d/10-sub-pixel-rgb.conf

which is pointing to:

/etc/fonts/conf.avail/10-sub-pixel-rgb.conf

Seems like the "append" still overrides the setting if the setting is "none":

$ cat /etc/fonts/conf.d/10-sub-pixel-rgb.conf
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<!-- Enable sub-pixel rendering -->
  <match target="pattern">
    <!--
      This configuration is available on the major desktop environments.
      We shouldn't overwrite it with "assign" unconditionally.
      Most clients may picks up the first value only. so using "append"
      may simply works to avoid it.
    -->
    <edit name="rgba" mode="append"><const>rgb</const></edit>
  </match>
</fontconfig>