TypeError: unhashable type

That's strange, the "unhashable type" is usually followed by which type it refers to. For example

TypeError: unhashable type: 'dict'

or

TypeError: unhashable type: 'list'

Can you give some steps to reproduce this error?

Xu Zhen: When this error appeared twitter web working properly?

This is probably being caused by data containing a dict instead of a list as expected.

I'm investigating what could cause this.

yes, it's a dict
I added "print data" before "del data[:]", and got these:

{u'user': {u'follow_request_sent': None, u'profile_use_background_image': True, u'default_profile_image': False, u'geo_enabled': False, u'verified': False, u'profile_image_url_https': u'***', u'profile_sidebar_fill_color': u'95E8EC', u'id': ***, u'profile_text_color': u'3C3940', u'followers_count': ***, u'profile_sidebar_border_color': u'***', u'id_str': u'***', u'profile_background_color': u'0099B9', u'listed_count': ***, u'utc_offset': 28800, u'statuses_count': ***, u'description': u'***', u'friends_count': ***, u'location': u'***', u'profile_link_color': u'0099B9', u'profile_image_url': u'***', u'notifications': None, u'show_all_inline_media': True, u'profile_background_image_url_https': u'https://si0.twimg.com/images/themes/theme4/bg.gif', u'profile_background_image_url': u'http://a1.twimg.com/images/themes/theme4/bg.gif', u'name': u'***', u'lang': u'***', u'following': None, u'profile_background_tile': False, u'favourites_count': ***, u'screen_name': u'***', u'url': u'***', u'created_at': u'***', u'contributors_enabled': False, u'time_zone': u'***', u'protected': False, u'default_profile': False, u'is_translator': False}, u'favorited': False, u'contributors': None, u'truncated': False, u'source': u'<a href="http://ifttt.com" rel="nofollow">ifttt</a>', u'text': u'***', u'created_at': u'***', u'retweeted': False, u'in_reply_to_status_id': None, u'coordinates': None, u'id': ***, u'entities': {u'user_mentions': [], u'hashtags': [], u'urls': [{u'indices': [9, 29], u'url': u'***', u'expanded_url': u'***', u'display_url': u'***'}]}, u'in_reply_to_status_id_str': None, u'place': None, u'in_reply_to_user_id': None, u'in_reply_to_screen_name': None, u'retweet_count': 0, u'geo': None, u'in_reply_to_user_id_str': None, u'possibly_sensitive': False, u'id_str': u'***'}

So for whatever reason the first data being processed by the buffer loop is not a list of ids as expect, but a tweet.

This is strange. I will investigate how this might happen. But of course it would be much easier if I myself could reproduce this.

Luis seems to be having this error consistently, all the time. How often does it happen to you, Xu?

this exception always occured.

I added "print self.queue" before "data, timestamp = self.queue.popleft()", and waited about 20 minutes

got those:
deque([([***, ***, ***], 1321280293.726731)])
deque([([***, ***, ... , ***], 1321280311.863793)])
deque([({u'delete': {u'status': {u'user_id': ***, u'user_id_str': u'***', u'id_str': u'***', u'id': ***}}}, 1321280319.168986)])
deque([([***, ***, ... , ***], 1321280423.949786)])
deque([({u'user': { ... }, 1321280430.24187)])
deque([({u'user': { ... }, 1321280441.315743)])
deque([({u'user': { ... }, 1321280477.380229)])

only the last one threw the TypeError exception

Were those lines printed without interruption, were there were disconnection messages between them?

Xu, can you print the entire output, not only the "print self.queue" output? This would help knowing the order of some events.

sometimes there were disconnect/reconnect messages, sometimes not.

this is an example output without any disconnect/reconnect messages in about 5 minutes.
when a tweet received, there will be a new "deque([({u'favorited': False, u'contributors': None, ... , u'in_reply_to_user_id': None}, 1234567890)])" message

Polly 0.93.0 (pre-alpha 3.0)
Copyright (C) 2011 Marcelo Hashimoto
cache path: /home/***/.cache/polly
config path: /home/***/.config/polly
data home: /home/***/.local/share/polly
initializing... done
deque([([***, ***, ... , ***], 1321335490.887468), ({u'favorited': False, u'contributors': None, ... , u'in_reply_to_user_id': None}, 1321335491.23738)])
DEBUG: *** connected successfully
deque([({u'favorited': False, u'contributors': None, ... , u'in_reply_to_user_id': None}, 1321335491.23738)])
deque([({u'favorited': False, u'contributors': None, ... , u'in_reply_to_user_id': None}, 1321335569.915518)])
deque([({u'favorited': False, u'contributors': None, ... , u'in_reply_to_user_id': None}, 1321335589.6702)])
deque([({u'favorited': False, u'contributors': None, ... , u'in_reply_to_user_id': None}, 1321335664.930679)])
deque([({u'favorited': False, u'contributors': None, ... , u'in_reply_to_user_id': None}, 1321335821.763521)])
DEBUG: *** disconnected successfully
finalizing... DEBUG: STREAM DELETED
DEBUG: ACCOUNT DELETED
done
clearing... done
Thank you for using Polly!
To contribute please visit https://launchpad.net/polly.

Can you post a full print like this one, but of a case where the bug happened?

I added some print statements in _buffer_loop() for the curl error 56 issue.

Polly 0.93.0 (pre-alpha 3.0)
Copyright (C) 2011 Marcelo Hashimoto
cache path: /home/***/.cache/polly
config path: /home/***/.config/polly
data home: /home/***/.local/share/polly
initializing... done
1st while loop
DEBUG: *** disconnected because of curl error 28
before self.buffer_thread.join()
for subdata in data:
break
2nd while loop
self.buffer_condition.acquire()
buffer_finished = self.buffer_finished
self.buffer_condition.release()
if buffer_finished:
break
after self.buffer_thread.join()
DEBUG: *** will reconnect in 0.25 seconds
1st while loop
1st while loop
for subdata in data:
break
2nd while loop
self.buffer_condition.acquire()
buffer_finished = self.buffer_finished
deque([([***, ***, ... , ***], 1321411242.826313)])
DEBUG: *** connected successfully
self.buffer_condition.release()
if buffer_finished:
2nd while loop
self.buffer_condition.acquire()
self.buffer_condition.wait()
for subdata in data:
break
2nd while loop
self.buffer_condition.acquire()
self.buffer_condition.wait()
buffer_finished = self.buffer_finished
deque([({u'favorited': False, u'in_reply_to_user_id': ***, ... , u'default_profile': True, u'is_translator': False}}, 1321411275.89721)])
self.buffer_condition.release()
if buffer_finished:
2nd while loop
self.buffer_condition.acquire()
self.buffer_condition.wait()
buffer_finished = self.buffer_finished
deque([({u'favorited': False, u'in_reply_to_user_id': ***, ... , u'default_profile': False, u'following': None}}, 1321411331.817188)])
Exception in thread Thread-54:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/dist-packages/polly/__init__.py", line 241, in run
    self.callback(*self.args)
  File "/usr/lib/python2.7/dist-packages/polly/twitter/realtime.py", line 437, in _buffer_loop
    del data[:]
TypeError: unhashable type

^CTraceback (most recent call last):
  File "/usr/bin/polly", line 332, in <module>
    Gtk.main()
KeyboardInterrupt

Thanks for the printout, I'll look into this when I have time.

Just want to verify with OP whether Polly is totally unusable after this error is encountered as I found in my system (see comment https://bugs.launchpad.net/polly/+bug/1254974/comments/4 in a possibly duplicate bug).

For the record, I didn't forget this report. But I couldn't reproduce the bug, couldn't think of a proper theory, and didn't receive any similar reports for two years, so it seems to happen on very specific conditions.

You shouldn't try things like adding a try/except or removing the line, because that would just sweep the issue under the rug. We need to figure out why dicts are reaching that part of the code, not skip them. I suggest you to print the data received so can look into that.

As a side note, can you explain how does one process a continuous stream of data, without blocking the GUI, without using threads? Also, does a race condition seems likely if you have a consistently reproducible test case?

I hid the last comment by mistake. Can be seen through this link:

https://bugs.launchpad.net/polly/+bug/887861/comments/16

Pasting comment 16 here, it's not available anymore at that link:

Tried properly debugging this time with some tracing:

Around this
https://bazaar.launchpad.net/~conscioususer/polly/unstable/view/head:/src/polly/twitter/realtime.py#L421

                if buffer_starting:
                    for subdata in data:
                        print u'DEBUG: reading 2nd subdata'
                        self.account.shelf.set_friend(subdata)
                    print u'DEBUG: deleting 2nd data[:]'
                    del data[:]

The first time del data[:] is called, it works. data is a list as expected in that case. The second time contents of data becomes a dict and we get:

DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: deleting 2nd data[:] <-------------- first call successful
DEBUG: k4rtik connected successfully

DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: deleting 2nd data[:] <-------------- second call error
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/site-packages/polly/__init__.py", line 241, in run
    self.callback(*self.args)
  File "/usr/lib/python2.7/site-packages/polly/twitter/realtime.py", line 423, in _buffer_loop
    del data[:]
TypeError: unhashable type

This is always reproducible. I tried adding try/except around del
dict[:] and removing del dict[:] altogether, but then another error
appears at same position:

DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: k4rtik connected successfully

DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
DEBUG: reading 2nd subdata
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/site-packages/polly/__init__.py", line 241, in run
    self.call...

Regarding the side issue first, I understand your point now. I think #python folks thought it was a command line client/library I was working with not a full desktop client, it was obvious from some of there later comments.

Might've found a clue while investigating another issue. It seems I am natively assuming that the first data received is always the friends list, but that does not seem to be always the case. I could certainly add some code to take this possibility into account.

Added a workaround to ignore non-list entries when the connection starts. Please test the latest trunk if possible.

Tried trunk, the symptom has gone - exception doesn't occur anymore. But my timeline doesn't load new tweets (except mentions) anymore as I pointed out in another bug at https://bugs.launchpad.net/polly/+bug/1254974/comments/4

Not sure where to put tracing code to debug this.

Is the connection established? Is the icon in the top left corner spinning? New tweets are not supposed to appear until it stops spinning. Check how much long it takes before it stops spinning.

It is established. I try the test after it has stopped spinning. First tweet goes fine, icon respins. Second tweet doesn't appear on Polly but gets posted, the icon doesn't respin this time. Third time also, similar behavior as second.

Interesting thing here, no new tweets appear for a while, but a mention to me appears. Then my response to my mention done from web interface also appears. Then I notice tweets appear occasionally (supposedly when it auto refreshes) but some tweets are missing in between. Attaching two screenshots to prove my point (1 and 2).

I think the problem starts the second time a tweet is posted from Polly. It doesn't refresh the timeline at that time, missing both the new tweet and any other tweets posted in between the time interval since first tweet.

Then it seems to be showing and not-showing a tweet alternately as shown in screenshots 3, 4 and 5.

Also, I get the following the in the terminal:

[k4rtik polly]$ polly
Polly 0.93.11 (pre-alpha 3.11)
Copyright (C) 2011 Marcelo Hashimoto
cache path: /home/kasingha/.cache/polly
config path: /home/kasingha/.config/polly
data home: /home/kasingha/.local/share/polly
initializing... done
DEBUG: k4rtik connected successfully
DEBUG: STREAM DELETED
DEBUG: STREAM DELETED
DEBUG: STREAM DELETED
?field.comment=It is established. I try the test after it has stopped spinning. First tweet goes fine, icon respins. Second tweet doesn't appear on Polly but gets posted, the icon doesn't respin this time. Third time also, similar behavior as second.

Interesting thing here, no new tweets appear for a while, but a mention to me appears. Then my response to my mention done from web interface also appears. Then I notice tweets appear occasionally (supposedly when it auto refreshes) but some tweets are missing in between. Attaching two screenshots to prove my point (1 and 2).

I think the problem starts the second time a tweet is posted from Polly. It doesn't refresh the timeline at that time, missing both the new tweet and any other tweets posted in between the time interval since first tweet.

Then it seems to be showing and not-showing a tweet alternately as shown in screenshots 3, 4 and 5.

Also, I get the following the in the terminal:

[k4rtik polly]$ polly
Polly 0.93.11 (pre-alpha 3.11)
Copyright (C) 2011 Marcelo Hashimoto
cache path: /home/kasingha/.cache/polly
config path: /home/kasingha/.config/polly
data home: /home/kasingha/.local/share/polly
initializing... done
DEBUG: k4rtik connected successfully
DEBUG: STREAM DELETED
DEBUG: STREAM DELETED
DEBUG: STREAM DELETED

Sorry, guess the comment got pasted again, while trying to attach the zip of screenshots.

It would be interesting if you could add a print in line 416 of realtime.py to check all data received.

Here's the trace, with a screenshot attached.

[k4rtik polly]$ polly
Polly 0.93.11 (pre-alpha 3.11)
Copyright (C) 2011 Marcelo Hashimoto
cache path: /home/kasingha/.cache/polly
config path: /home/kasingha/.config/polly
data home: /home/kasingha/.local/share/polly
initializing... done
[1657730988, 32468665, 49793, 4620451, 21378183, 2170941817, 76612335, 1183684093, 133332145, 31353077, 15390117, 14510158, 16521996, 13479072, 21010243, 71845035, 14278978, 17717614, 44132827, 14452322, 2999841, 18582812, 13334762, 84555119, 752673, 176991766, 34670660, 14685475, 24705126, 49527510, 243112674, 141111177, 14362098, 58326841, 15300300, 58560480, 93473293, 15804774, 90871277, 10954532, 147493686, 404472157, 1379453478, 17703462, 41741713, 9533042, 21295041, 14197826, 131498466, 255291612, 5377742, 23971219, 14978981, 17914570, 12150972, 102022974, 19177581, 405427035, 42573217, 484757080, 14839109, 103770785, 18130823, 162776372, 18768244, 7156792, 724473, 3839, 993006330, 42018126, 31181347, 2098451, 16004473, 11333192, 16944211, 221825617, 187667133, 14057362, 15098340, 45321924, 15439395, 62474515, 18987429, 14561327, 39465515, 56505125, 91458830, 15335523, 63049749, 14679848, 18393773, 808009, 15405349, 35104393, 77827772, 41285641, 123621072, 69279691, 93587015, 67890042, 2384071, 243284052, 69893, 17522755, 457886651, 87388484, 84351228, 50254941, 19160166, 60310843, 14423566, 22492607, 8822402, 8633952, 52964853, 19071667, 3151421, 11647412, 135120796, 292765643, 18040514, 126881128, 17777667, 7788062, 102982202, 17861792, 75714954, 1666038950, 20536157, 51845510, 16589206, 283536994, 59178044, 75478733, 227363436, 49324568, 15421904, 14145498, 17186020, 471741741, 68627219, 143257937, 879114595, 8385342, 16665197, 25037233, 935021, 101311381, 2789291, 14471155, 54714299, 5637652, 93957809, 84118228, 15286109, 5502392, 142394063, 148265382, 77941245, 1665621332, 611986351, 97865628, 186154646, 14249789, 13567, 1962506023, 318063815, 9155142, 29370023, 12526082, 12904842, 8207832, 176864693, 43568964, 939053190, 281015822, 7876572, 783214, 1031966198, 824065, 65289126, 67914616, 14369310, 277434037, 146186456, 17484680, 10385792, 50478950, 5562702, 156897798, 61453726, 14637812, 7144422, 1547468708, 86390214, 255617445, 28360450, 16936052, 111889642, 72525598, 44067193, 163952496, 139618840, 356567464, 56950834, 15074679, 1183041, 13621292, 1039443116, 23269231, 100505040, 326585552, 15649930, 15865677, 14961265, 7839952, 19575586, 44196397, 15866192, 14443876, 1567548523, 18080969, 18839785, 267265444, 641003, 662693, 15492359, 572034749, 220139885, 38873536, 18137723, 14214962, 16387631, 105083, 17232315, 18755782, 814641, 1671811, 18761671, 43592062, 16838153, 64719766, 15347534, 14300782, 19816326, 70404281, 7542902, 7375062, 19018183, 17257494, 2425151, 42374574, 72915446, 15948437, 53720690, 93711247, 17485879, 16578411, 17482727, 382267114, 466563654, 304944155, 713263, 16255384, 73879067, 7109592, 15859268, 223807771, 2142731, 13679632, 1908450804, 149505972, 37179280, 19814653, 6467102, 14304568, 16532702, 50393960, 39383125, 14761795, 1076217319, 79747253, 48318008, 85010311, 176376937, 9980812, 38679388, 5520952, 747153, 68977380,...

Well, the data seems correct. Now can you check whether the corresponding data is printed as expected when you post/delete a tweet or are supposed to receive a new one?

Yes, the data is printed as expected in case of posting a new tweet or deleting one. But it seems the stream fails to refresh automatically after I post a new tweet (except before the first tweet posted). In the following trace, I first posted 2 tweets, only the first one appeared on Polly timeline, then I deleted the first one, then posted two more tweets, but only the last one (4th) appeared on Polly timeline.

Looks like something is breaking the auto-refresh code after the first tweet is posted successfully. Should probably put some tracing around that code as well.

[k4rtik polly]$ polly
Polly 0.93.11 (pre-alpha 3.11)
Copyright (C) 2011 Marcelo Hashimoto
cache path: /home/kasingha/.cache/polly
config path: /home/kasingha/.config/polly
data home: /home/kasingha/.local/share/polly
initializing... done
[37208728, 1657730988, 32468665, 49793, 4620451, 21378183, 2170941817, 76612335, 1183684093, 133332145, 31353077, 15390117, 14510158, 16521996, 13479072, 21010243, 71845035, 14278978, 17717614, 44132827, 14452322, 2999841, 18582812, 13334762, 84555119, 752673, 176991766, 34670660, 14685475, 24705126, 49527510, 243112674, 141111177, 14362098, 58326841, 15300300, 58560480, 93473293, 15804774, 90871277, 10954532, 147493686, 21772336, 404472157, 1379453478, 17703462, 41741713, 9533042, 21295041, 14197826, 131498466, 255291612, 5377742, 23971219, 14978981, 17914570, 12150972, 102022974, 19177581, 405427035, 42573217, 484757080, 14839109, 103770785, 18130823, 162776372, 18768244, 7156792, 724473, 3839, 993006330, 42018126, 31181347, 2098451, 16004473, 11333192, 16944211, 221825617, 187667133, 14057362, 15098340, 45321924, 15439395, 62474515, 18987429, 14561327, 39465515, 56505125, 91458830, 15335523, 63049749, 14679848, 18393773, 808009, 15405349, 35104393, 77827772, 41285641, 123621072, 69279691, 93587015, 67890042, 2384071, 243284052, 69893, 17522755, 457886651, 87388484, 84351228, 50254941, 19160166, 75974281, 60310843, 14423566, 22492607, 8822402, 8633952, 52964853, 19071667, 3151421, 11647412, 135120796, 292765643, 18824526, 18040514, 126881128, 17777667, 7788062, 102982202, 17861792, 75714954, 1666038950, 20536157, 51845510, 16589206, 283536994, 59178044, 75478733, 227363436, 49324568, 15421904, 14145498, 17186020, 471741741, 68627219, 143257937, 879114595, 8385342, 16665197, 25037233, 935021, 101311381, 2789291, 14471155, 54714299, 5637652, 93957809, 84118228, 15286109, 5502392, 142394063, 148265382, 77941245, 1665621332, 611986351, 97865628, 186154646, 14249789, 13567, 1962506023, 318063815, 9155142, 29370023, 12526082, 12904842, 8207832, 176864693, 43568964, 939053190, 281015822, 7876572, 783214, 1031966198, 824065, 65289126, 67914616, 14369310, 277434037, 146186456, 17484680, 10385792, 50478950, 5562702, 156897798, 61453726, 17939037, 14637812, 7144422, 1547468708, 86390214, 255617445, 28360450, 16936052, 111889642, 72525598, 44067193, 163952496, 139618840, 356567464, 56950834, 15074679, 1183041, 13621292, 1039443116, 23269231, 100505040, 326585552, 15649930, 15865677, 14961265, 7839952, 19575586, 44196397, 15866192, 14443876, 15143478, 1567548523, 18080969, 18839785, 56304605, 267265444, 641003, 14835908, 662693, 15492359, 572034749, 22...

So if I understood correctly *everything* was printed, but some were not rendered in the GUI?

Exactly.