HPLIP

Hplip does not work with Python3: Unicode error

Bug #1480152 reported by Johan Myréen on 2015-07-31

This bug affects 5 people

	Status	Importance	Assigned to
HPLIP	New	Undecided	Unassigned
hplip (Arch Linux)	New	Undecided	Unassigned
hplip (Debian)	Fix Released	Unknown	debbugs #861731
hplip (Fedora)	Confirmed	Undecided	redhat-bugs #1573430
hplip (Gentoo Linux)	Fix Released	Medium	gentoo-bugs #651870

Bug Description

The hp-setup program crashes with a UnicodeDecodeError when it tries to detect my Photosmart 2575 printer. I am running Arch Linux which has switched to using Python3 by default (/usr/bin/python is a symlink to python3).

Below is a part of a debugging session demonstrating the problem. The variable x contains (part of) the response from my printer to the multicast query sent on the network. The to_string_utf8 function is supposed to convert the contents of the byte array to UTF-8, but chokes on the 0xf0 in position 364, presumably because the bytes aren't all strictly ASCII codepoints.

Running the program using Python2 does not trigger this bug.

My printer is a HP Photosmart 2575, Python version 3.4.3, Python2 version 2.7.10, Hplip version hplip 3.15.6-1 (Arch Linux).

(Pdb) n
> /usr/share/hplip/base/slp.py(110)detectNetworkDevices()
-> struct.unpack("!BBHBBHHHHH", data[:16])
(Pdb) n
> /usr/share/hplip/base/slp.py(112)detectNetworkDevices()
-> x = struct.unpack("!%ds" % attr_length, data[16:])[0].strip()
(Pdb) n
> /usr/share/hplip/base/slp.py(115)detectNetworkDevices()
-> x= to_string_utf8(x)
(Pdb) p x
b'(x-hp-ver=01)(x-hp-mac=00187100690d)(x-hp-num_port=01)(x-hp-ip=192.168.000.102)(x-hp-hn=Smarty)(x-hp-p1=MFG:HP;MDL:Photosmart 2570 series;CMD:MLC,PCL,PML,DW-PCL,DESKJET,DYN;1284.4DL:4d,4e,1;CLS:PRINTER;DES:6543;SN:MY66U1131C04DY;S:038000C484001021002c1800028c2880025;J: ;Z:0102,0503dea9015549,0600;BT:000000000000,50686F746F736D6172742032353\xd1\xbdZT\xf0!\x80i \x82\xc8,\xff\xff\xfa@\xf0,\x85-)(x-hp-guid=00187100690d)'
(Pdb) n
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 364: invalid continuation byte
> /usr/share/hplip/base/slp.py(115)detectNetworkDevices()
-> x= to_string_utf8(x)
(Pdb)

Revision history for this message

Gaurav Sood (gaurav-sood) wrote on 2015-07-31:

sixext.patch Edit (395 bytes, text/plain)

Please apply this patch and see if it works
1. Take a back up of /usr/share/hplip/base/sixext.py
1. Download this patch in any folder and navigate to that folder
2. Apply this patch as a root user using the following command

'patch /usr/share/hplip/base/sixext.py < sixext.patch'

Revision history for this message

Johan Myréen (r-ubuntuone-v) wrote on 2015-07-31:

Yes, now the decode functions ignore the errors and the setup succeeds.

Thank you.

Revision history for this message

Georges (sim0nx) wrote on 2017-05-03:

The latest Debian Stretch version is affected by this bug, which is version 3.16.11+repack0-2.

I ended up patching the "/usr/share/hplip/base/sixext.py" file exactly as in the suggested patch, which solved the issue for us.

Is there any information on when this quite old fix will make it into the upstream package?

Georges (sim0nx) on 2017-05-03

Changed in hplip (Debian):
importance:	Undecided → Unknown
status:	New → Unknown

Bug Watch Updater (bug-watch-updater) on 2017-05-03

Changed in hplip (Debian):
status:	Unknown → Confirmed

Revision history for this message

kritika arora (kritika-arora1) wrote on 2017-05-04:

Hi,

The required changes has been included in HPLIP source and will be made available in the next release.

Thanks.

Changed in hplip:
status:	New → Fix Committed

Revision history for this message

Georges (sim0nx) wrote on 2017-05-05:

Hi,

Thank you!

Bug Watch Updater (bug-watch-updater) on 2017-05-05

Changed in hplip (Debian):
status:	Confirmed → Fix Committed

Bug Watch Updater (bug-watch-updater) on 2017-05-08

Changed in hplip (Debian):
status:	Fix Committed → Fix Released

Revision history for this message

Daniel Pielmeier (daniel-pielmeier) wrote on 2018-04-08:

According to #4 this has been fixed, however the same issue came up in Gentoo recently. Applying the patch from #1 fixed the issue.

What could be the problem here. Was this fix reverted or changed in another way. Maybe there is a regression somewhere.

Bug Watch Updater (bug-watch-updater) on 2018-04-08

Changed in hplip (Gentoo Linux):
importance:	Unknown → Medium
status:	Unknown → New

Bug Watch Updater (bug-watch-updater) on 2018-04-21

Changed in hplip (Gentoo Linux):
status:	New → Fix Released

Francois Jaouen (francois-jaouen) on 2018-04-30

Changed in hplip:
status:	Fix Committed → New

Revision history for this message

Francois Jaouen (francois-jaouen) wrote on 2018-04-30:

full output of : hp-probe -bnet -ldebug Edit (3.4 KiB, text/plain)

Tested version 3.18.4 on Arch, the problem is still there. The proposed patch has not been applied :

% hp-probe -bnet
HP Linux Imaging and Printing System (ver. 3.18.4)
(...)
\Traceback (most recent call last):
  File "/usr/bin/hp-probe", line 162, in <module>
    devices = device.probeDevices([b], timeout, ttl, filter_dict, search, method)
  File "/usr/share/hplip/base/device.py", line 487, in probeDevices
    detected_devices = slp.detectNetworkDevices(ttl, timeout)
  File "/usr/share/hplip/base/slp.py", line 115, in detectNetworkDevices
    x= to_string_utf8(x)
  File "/usr/share/hplip/base/sixext.py", line 113, in to_string_utf8
    return s.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 365-366: invalid continuation byte

The printer is an HP Photosmart C6480 All in One.

Revision history for this message

Francois Jaouen (francois-jaouen) wrote on 2018-04-30:

Look like a duplicate of https://bugs.launchpad.net/hplip/+bug/1473066

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

Download full text (5.0 KiB)

Description of problem:
Try to detect my network printer HP Photosmart C8260 using the command :
% hp-probe -bnet -ldebug

Output of the command :

HP Linux Imaging and Printing System (ver. 3.18.4)
Printer Discovery Utility ver. 4.1

Copyright (c) 2001-15 HP Development Company, LP
This software comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to distribute it
under certain conditions. See COPYING file for more details.

--------------------
| DEVICE DISCOVERY |
--------------------

Probing network for printers. Please wait, this will take approx. 10 seconds...

hp-probe[1859]: debug: {}
hp-probe[1859]: debug: 0000: 01 07 01 a7 00 00 65 6e 00 03 00 00 00 00 01 97 28 78 2d 68 70 2d 76 65 72 3d 30 31 29 28 78 2d ......en........(x-hp-ver=01)(x-
hp-probe[1859]: debug: 0020: 68 70 2d 6d 61 63 3d 30 30 31 65 30 62 39 61 35 63 63 63 29 28 78 2d 68 70 2d 6e 75 6d 5f 70 6f hp-mac=001e0b9a5ccc)(x-hp-num_po
hp-probe[1859]: debug: 0040: 72 74 3d 30 31 29 28 78 2d 68 70 2d 69 70 3d 31 39 32 2e 31 36 38 2e 32 34 36 2e 30 31 31 29 28 rt=01)(x-hp-ip=192.168.246.011)(
hp-probe[1859]: debug: 0060: 78 2d 68 70 2d 68 6e 3d 48 50 39 41 35 43 43 43 29 28 78 2d 68 70 2d 70 31 3d 4d 46 47 3a 48 50 x-hp-hn=HP9A5CCC)(x-hp-p1=MFG:HP
hp-probe[1859]: debug: 0080: 3b 4d 44 4c 3a 50 68 6f 74 6f 73 6d 61 72 74 20 43 36 32 30 30 20 73 65 72 69 65 73 3b 43 4d 44 ;MDL:Photosmart C6200 series;CMD
hp-probe[1859]: debug: 00a0: 3a 4d 4c 43 2c 50 43 4c 2c 50 4d 4c 2c 44 57 2d 50 43 4c 2c 44 45 53 4b 4a 45 54 2c 44 59 4e 3b :MLC,PCL,PML,DW-PCL,DESKJET,DYN;
hp-probe[1859]: debug: 00c0: 31 32 38 34 2e 34 44 4c 3a 34 64 2c 34 65 2c 31 3b 43 4c 53 3a 50 52 49 4e 54 45 52 3b 44 45 53 1284.4DL:4d,4e,1;CLS:PRINTER;DES
hp-probe[1859]: debug: 00e0: 3a 43 43 39 38 38 3b 53 4e 3a 4d 59 37 43 49 48 35 31 30 50 30 35 30 36 3b 53 3a 30 33 38 30 38 :CC988;SN:MY7CIH510P0506;S:03808
hp-probe[1859]: debug: 0100: 38 43 34 38 34 30 30 31 30 32 31 30 30 37 38 63 62 30 30 30 30 30 34 31 62 38 30 34 32 37 34 36 8C4840010210078cb0000041b8042746
hp-probe[1859]: debug: 0120: 62 38 30 34 34 38 34 37 62 38 30 34 34 65 34 34 62 38 30 34 36 34 34 38 62 38 30 30 35 35 34 35 b8044847b8044e44b8046448b8005545
hp-probe[1859]: debug: 0140: 62 38 30 34 36 34 3b 5a 3a 30 31 30 32 2c 30 35 30 30 30 30 30 39 30 31 36 61 38 35 30 31 33 32 b80464;Z:0102,05000009016a850132
hp-probe[1859]: debug: 0160: 34 35 30 31 33 32 34 35 30 31 33 32 34 35 30 31 33 32 34 31 30 31 33 32 34 35 21 73 54 f0 a0 29 45013245013245013241013245!sT..)
hp-probe[1859]: debug: 0180: 4d 7d 21 73 55 ac ff ff fa 40 a0 39 c7 3f 29 28 78 2d 68 70 2d 67 75 69 64 3d 30 30 31 65 30 62 M}!sU....@.9.?)(x-hp-guid=001e0b
hp-probe[1859]: debug: 01a0: 39 61 35 63 63 63 29 9a5ccc)
Traceback (most recent call last):
  File "/usr/bin/hp-probe", line 162, in <module>
    devices = device.probeDevices([b], timeout, ttl, filter_dict, search, method)
  File "/usr/share/hplip/base/device.py", line 487, in probeDevices
    detected_devices = slp.detectNetworkDevices(ttl, timeout)
  File "/usr/share/hplip/base/slp.py", line 115, in detectNe...

Description of problem:
Try to detect my network printer HP Photosmart C8260 using the command :
 % hp-probe -bnet -ldebug

Output of the command :

HP Linux Imaging and Printing System (ver. 3.18.4)
Printer Discovery Utility ver. 4.1

--------------------
| DEVICE DISCOVERY |
--------------------

Probing network for printers. Please wait, this will take approx. 10 seconds...

hp-probe[1859]: debug: {}
hp-probe[1859]: debug: 0000:  01 07 01 a7 00 00 65 6e 00 03 00 00 00 00 01 97 28 78 2d 68 70 2d 76 65 72 3d 30 31 29 28 78 2d  ......en........(x-hp-ver=01)(x-
hp-probe[1859]: debug: 0020:  68 70 2d 6d 61 63 3d 30 30 31 65 30 62 39 61 35 63 63 63 29 28 78 2d 68 70 2d 6e 75 6d 5f 70 6f  hp-mac=001e0b9a5ccc)(x-hp-num_po
hp-probe[1859]: debug: 0040:  72 74 3d 30 31 29 28 78 2d 68 70 2d 69 70 3d 31 39 32 2e 31 36 38 2e 32 34 36 2e 30 31 31 29 28  rt=01)(x-hp-ip=192.168.246.011)(
hp-probe[1859]: debug: 0060:  78 2d 68 70 2d 68 6e 3d 48 50 39 41 35 43 43 43 29 28 78 2d 68 70 2d 70 31 3d 4d 46 47 3a 48 50  x-hp-hn=HP9A5CCC)(x-hp-p1=MFG:HP
hp-probe[1859]: debug: 0080:  3b 4d 44 4c 3a 50 68 6f 74 6f 73 6d 61 72 74 20 43 36 32 30 30 20 73 65 72 69 65 73 3b 43 4d 44  ;MDL:Photosmart C6200 series;CMD
hp-probe[1859]: debug: 00a0:  3a 4d 4c 43 2c 50 43 4c 2c 50 4d 4c 2c 44 57 2d 50 43 4c 2c 44 45 53 4b 4a 45 54 2c 44 59 4e 3b  :MLC,PCL,PML,DW-PCL,DESKJET,DYN;
hp-probe[1859]: debug: 00c0:  31 32 38 34 2e 34 44 4c 3a 34 64 2c 34 65 2c 31 3b 43 4c 53 3a 50 52 49 4e 54 45 52 3b 44 45 53  1284.4DL:4d,4e,1;CLS:PRINTER;DES
hp-probe[1859]: debug: 00e0:  3a 43 43 39 38 38 3b 53 4e 3a 4d 59 37 43 49 48 35 31 30 50 30 35 30 36 3b 53 3a 30 33 38 30 38  :CC988;SN:MY7CIH510P0506;S:03808
hp-probe[1859]: debug: 0100:  38 43 34 38 34 30 30 31 30 32 31 30 30 37 38 63 62 30 30 30 30 30 34 31 62 38 30 34 32 37 34 36  8C4840010210078cb0000041b8042746
hp-probe[1859]: debug: 0120:  62 38 30 34 34 38 34 37 62 38 30 34 34 65 34 34 62 38 30 34 36 34 34 38 62 38 30 30 35 35 34 35  b8044847b8044e44b8046448b8005545
hp-probe[1859]: debug: 0140:  62 38 30 34 36 34 3b 5a 3a 30 31 30 32 2c 30 35 30 30 30 30 30 39 30 31 36 61 38 35 30 31 33 32  b80464;Z:0102,05000009016a850132
hp-probe[1859]: debug: 0160:  34 35 30 31 33 32 34 35 30 31 33 32 34 35 30 31 33 32 34 31 30 31 33 32 34 35 21 73 54 f0 a0 29  45013245013245013241013245!sT..)
hp-probe[1859]: debug: 0180:  4d 7d 21 73 55 ac ff ff fa 40 a0 39 c7 3f 29 28 78 2d 68 70 2d 67 75 69 64 3d 30 30 31 65 30 62  M}!sU....@.9.?)(x-hp-guid=001e0b
hp-probe[1859]: debug: 01a0:  39 61 35 63 63 63 29                                                                             9a5ccc)
Traceback (most recent call last):
  File "/usr/bin/hp-probe", line 162, in <module>
    devices = device.probeDevices([b], timeout, ttl, filter_dict, search, method)
  File "/usr/share/hplip/base/device.py", line 487, in probeDevices
    detected_devices = slp.detectNetworkDevices(ttl, timeout)
  File "/usr/share/hplip/base/slp.py", line 115, in detectNetworkDevices
    x= to_string_utf8(x)
  File "/usr/share/hplip/base/sixext.py", line 113, in to_string_utf8
    return s.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 365-366: invalid continuation byte

Version-Release number of selected component:
hplip-3.18.4-1.fc28

Additional info:
reporter:       libreport-2.9.5
cmdline:        /usr/bin/python3 /usr/bin/hp-probe -bnet -ldebug
crash_function: to_string_utf8
exception_type: UnicodeDecodeError
executable:     /usr/bin/hp-probe
interpreter:    python3-3.6.5-1.fc28.x86_64
kernel:         4.16.5-300.fc28.x86_64
runlevel:       N 5
type:           Python3
uid:            1000

Truncated backtrace:
sixext.py:113:to_string_utf8:UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 365-366: invalid continuation byte

Traceback (most recent call last):
  File "/usr/bin/hp-probe", line 162, in <module>
    devices = device.probeDevices([b], timeout, ttl, filter_dict, search, method)
  File "/usr/share/hplip/base/device.py", line 487, in probeDevices
    detected_devices = slp.detectNetworkDevices(ttl, timeout)
  File "/usr/share/hplip/base/slp.py", line 115, in detectNetworkDevices
    x= to_string_utf8(x)
  File "/usr/share/hplip/base/sixext.py", line 113, in to_string_utf8
    return s.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 365-366: invalid continuation byte

Local variables in innermost frame:
s: b'(x-hp-ver=01)(x-hp-mac=001e0b9a5ccc)(x-hp-num_port=01)(x-hp-ip=192.168.246.011)(x-hp-hn=HP9A5CCC)(x-hp-p1=MFG:HP;MDL:Photosmart C6200 series;CMD:MLC,PCL,PML,DW-PCL,DESKJET,DYN;1284.4DL:4d,4e,1;CLS:PRINTER;DES:CC988;SN:MY7CIH510P0506;S:038088C4840010210078cb0000041b8042746b8044847b8044e44b8046448b8005545b80464;Z:0102,05000009016a85013245013245013245013241013245!sT\xf0\xa0)M}!sU\xac\xff\xff\xfa@\xa09\xc7?)(x-hp-guid=001e0b9a5ccc)'

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#10

Created attachment 1429138
File: backtrace

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#11

Created attachment 1429139
File: cgroup

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#12

Created attachment 1429140
File: cpuinfo

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#13

Created attachment 1429141
File: environ

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#14

Created attachment 1429142
File: mountinfo

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#15

Created attachment 1429143
File: namespaces

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#16

Created attachment 1429144
File: open_fds

Revision history for this message

In Red Hat Bugzilla #1573430, francois.jaouen (francois.jaouen-redhat-bugs) wrote on 2018-05-01:

#17

Created attachment 1429145
File: hp-probe.log

Revision history for this message

In Red Hat Bugzilla #1573430, johnthacker (johnthacker-redhat-bugs) wrote on 2019-02-10:

#18

Created attachment 1528767
Patch provided in Launchpad bug

Patch for /usr/share/hplip/base/sixext.py provided in the Launchpad bug for the same issue.

Revision history for this message

In Red Hat Bugzilla #1573430, johnthacker (johnthacker-redhat-bugs) wrote on 2019-02-10:

#19

This is exposed by the switch to using Python3 by default. It's a known bug in HPlip that other distributions have seen when they switched to Python3 by default. It is still present in Fedora 29 and the new HPlip. There's a patch that makes things work (but is probably not the Right Way) by ignoring the UTF-8 conversion errors. Using the patch, everything works for me (and people on other distros.)

Bug Watch Updater (bug-watch-updater) on 2019-02-10