'TypeError: coercing to Unicode' on smart update

msg1332 (view) Author: ed Date: 2008-04-30.10:22:35

Here is a different occurrence of the same error (note the different call stack).

# smart install lrzip
Loading cache...
Updating cache... ######################################## [100%]

Computing transaction...

Installing packages (1):
  lrzip-0.19-1.fc9@i386

75.5kB of package files are needed. 75.5kB will be used.

Confirm changes? (Y/n): y

Fetching packages...
-> http://download.fedora.redhat.com/pub/fedora/.../lrzip-0.19-1.fc9.i386.rpm
lrzip-0.19-1.fc9.i386.rpm ( 0%)

Committing transaction...
Traceback (most recent call last):
  File "/usr/bin/smart", line 194, in <module>
    main(sys.argv[1:])
  File "/usr/bin/smart", line 167, in main
    exitcode = iface.run(opts.command, opts.argv)
  File "/usr/lib/python2.5/site-packages/smart/interface.py", line 53, in run
    result = _command.main(self._ctrl, opts)
  File "/usr/lib/python2.5/site-packages/smart/commands/install.py", line 178,
in main
    ctrl.commitTransaction(trans, confirm=confirm)
  File "/usr/lib/python2.5/site-packages/smart/control.py", line 500, in
commitTransaction
    return self.commitChangeSet(trans.getChangeSet(), caching, confirm)
  File "/usr/lib/python2.5/site-packages/smart/control.py", line 558, in
commitChangeSet
    pmclass().commit(pmcs, pkgpaths)
  File "/usr/lib/python2.5/site-packages/smart/backends/rpm/pm.py", line 155, in
commit
    fd = os.open(path, os.O_RDONLY)
TypeError: coercing to Unicode: need string or buffer, NoneType found
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:

msg1254 (view) Author: ed Date: 2007-12-04.16:34:40

I am now able to reproduce this at the console.

root@localhost ~ # smart update
Loading cache...
Updating cache... ######################################## [100%]

Fetching information for 'Fedora 8 - i386'...
-> http://download.fedora.redhat.com/pub/fedora/linux/releases/8/.../repomd.xml
Traceback (most recent call last): ( 0%)
  File "/usr/bin/smart", line 194, in <module>
    main(sys.argv[1:])
  File "/usr/bin/smart", line 167, in main
    exitcode = iface.run(opts.command, opts.argv)
  File "/usr/lib/python2.5/site-packages/smart/interface.py", line 53, in run
    result = _command.main(self._ctrl, opts)
  File "/usr/lib/python2.5/site-packages/smart/commands/update.py", line 78, in main
    failed = not ctrl.reloadChannels(channels, caching=NEVER)
  File "/usr/lib/python2.5/site-packages/smart/control.py", line 352, in
reloadChannels
    if not channel.fetch(self._fetcher, progress):
  File "/usr/lib/python2.5/site-packages/smart/channels/rpm_md.py", line 73, in
fetch
    digest = getFileDigest(item.getTargetPath())
  File "/usr/lib/python2.5/site-packages/smart/util/filetools.py", line 31, in
getFileDigest
    file = open(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:

msg1239 (view) Author: ed Date: 2007-11-15.15:15:25

When running 'smart update' I saw this error:

Loadin...

I got the same error as well - Sequence of events

# smart update
/usr/lib64/python2.6/site-packages/smart/plugins/zyppchannelsync.py:22: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
Loading cache...
Updating cache... ########################################################################## [100%]

Fetching information for 'kde3'...
-> http://ftp5.gwdg.de/pub/opensuse/repositories/KDE:/KDE3/openSUSE_11.1/repodata/repomd.xml
repomd.xml ########################################################################## [ 5%]

Fetching information for 'factory-oss'...
-> http://mirror.internode.on.net/pub/opensuse/factory/repo/oss/media.1/media
media ########################################################################## [ 15%]
-> http://mirror.internode.on.net/pub/opensuse/factory/repo/oss/content
content ########################################################################## [ 20%]
-> http://mirror.internode.on.net/pub/opensuse/factory/repo/oss/suse/setup/descr/packages.gz
packages.gz ########################################################################## [ 25%]
-> http://mirror.internode.on.net/pub/opensuse/factory/repo/oss/suse/setup/descr/packages.en.gz
packages.en.gz ########################################################################## [ 30%]

Fetching information for 'factory-non-oss'...
-> http://mirror.internode.on.net/pub/opensuse/factory/repo/non-oss/media.1/media
media ########################################################################## [ 35%]
                                                                                                                             ...

I'm running Fedora 10, x86_64 and occasionally see the 'Coercing to unicode' crash when attempting to download repo data. I don't know what causes it, but suspect it may be linked to repo updates. After 'kill -9' and a restart smart usually works as normal. Here's a backtrace, submitted at Axel's suggestion after I reported the problem on the ATrpms-users' list.

http://www.gossamer-threads.com/lists/atrpms/users/12895
----------------------
Traceback (most recent call last):

  File "/usr/lib64/python2.5/site-packages/smart/interfaces/gtk/interactive.py", line 180, in callback
    exec code in globals

  File "<callback>", line 1, in <module>

  File "/usr/lib64/python2.5/site-packages/smart/interfaces/gtk/interactive.py", line 411, in updateChannels
    self._ctrl.reloadChannels(channels, caching=NEVER)

  File "/usr/lib64/python2.5/site-packages/smart/control.py", line 359, in reloadChannels
    if not channel.fetch(self._fetcher, progress):

  File "/usr/lib64/python2.5/site-packages/smart/channels/rpm_md.py", line 115, in fetch
    digest = getFileDigest(item.getTargetPath())

  File "/usr/lib64/python2.5/site-packages/smart/util/filetools.py", line 34, in getFileDigest
    file = open(path)

TypeError: coercing to Unicode: need string or buffer, NoneType found
 --------------------

Sometimes Fetcher._uncompress() invoked in a new thread from Fetcher.run() finished later than Fetcher.run() and item.setFailed()/item.setSucceeded() wasn't yet called because of that. And because item._targetpath is set at the end of these methods, getFileDigest() had nothing to calculate digest from.

Possible fix in lp:~dsmid/smart/coercing2unicode

Fetcher.run shouldn't finish before self._uncompressing is 0 ? Does it need a lock perhaps ?

Otherwise this should probably be done in the Fetcher (or FetchItem) classes, with a method.

Uncompressing should finish before item.getStatus() or item.getTargetPath() needs to be called. These attributes are not guaranteed to be populated when Fetcher.run() finishes.
If this is generally not acceptable, then Fetcher.run() shouldn't finish before uncompressing is done and it should wait for uncompressing child threads to finish.

What I meant was that I would have thought that it should wait already ? It does self._uncompressing += 1 before starting the thread and self._uncompressing -= 1 after setSucceeded/setFailed.

                del self._active[handle]
                userhost = (url.user, url.host, url.port)
                self._inactive[handle] = userhost

                if http_code == 404:
                    item.setFailed(_("File not found"))
                else:
                    valid, reason = fetcher.validate(item, localpath,
                                                     withreason=True)
                    if valid:
                        fetchedsize = handle.getinfo(pycurl.SIZE_DOWNLOAD)
                        item.setSucceeded(localpath, fetchedsize)
                    elif handle.partsize:
                        self._queue.append(item)
                    else:
                        item.setFailed(reason)

Sorry about the previous comment, it was a misclick. Can anybody remove it ?

You're right, the condition "active or self._uncompressing" won't let Fetcher.run() finish if there are some files being uncompressed.

Anyway, Fetcher.run() somehow returns without calling neither item.setSucceeded() nor item.setFailed().
In URLLIBHandler.tick() there's a moment where race conditions can cause bad thinks to happen:

       self._lock.release()
# <== What if URLLIBHandler.fetch() in a concurrent thread pops self._queue here ?
       return bool(self._queue or self._active)

I witnessed tick() starting five fetch() threads for downloading just one file. Apparently the first one that pops self._queue wins and the others die and decrement self._active almost immediately but I still consider it a bug.