LANG is explicitly set to C, but some services (like glance) want to read files with utf8 characters

Bug #1697733 reported by Clark Boylan
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fix Released
Sean Dague

Bug Description

glance-manage throws errors under the python3.5 job because it attempts to open and read a file with utf8 characters in it, but devstack has hard set LANG=C.

ERROR glance.db.sqlalchemy.metadata [-] Failed to parse json file /etc/glance/metadefs/compute-trust.json while populating metadata due to: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not in range(128): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 90: ordinal not in range(128)

This only happens under python3 because python3 open() will refer to locale.getpreferredencoding() by default if no encoding is explicitly set. Python2 doesn't have this problem because strings and open operate on binary not encoded things.

Devstack sets LANG=C at:

Example job run where this happens:

One thing that makes this tricky is that open() under python2 doesn't take an encoding while open() under python3 does. Easy enough to handle this in code but maybe we should try and get six to address this?

Also worth noting that the infra test nodes should have a locale of C.utf8 or C.UTF-8, but these locales are apparently (not yet) universal.

Considering that devstack wants to enforce and ascii locale the simplest option here may just be to remove the utf8 characters from the metadata json files. '®' and '–' are the two characters which can be replaced with '(R)' and '-'.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to devstack (master)

Fix proposed to branch: master

Changed in devstack:
assignee: nobody → Clark Boylan (cboylan)
status: New → In Progress
Changed in devstack:
assignee: Clark Boylan (cboylan) → Sean Dague (sdague)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Submitter: Jenkins
Branch: master

commit d095e97624467fb1e0fa38955b45960d3cbc5651
Author: Clark Boylan <email address hidden>
Date: Tue Jun 13 10:18:36 2017 -0700

    Support unicode via en_US.utf8

    Because C.utf8 is not everywhere and is sometimes called C.UTF-8 (just
    to confuse people) use en_US.utf8 which is in most places. This isn't
    language/region agnostic but gives a consistent unicode aware locale to

    Change-Id: I67a8c77a5041e9cee740adf0e02fdc9b183c5bc4
    fixes-bug: 1697733

Changed in devstack:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.