MySQL Connector/Python

connection.py fails on incomplete packet send

Bug #711520 reported by David Creemer
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL Connector/Python
Status tracked in Trunk
0.3
Fix Committed
High
Geert JM Vanderkelen
Trunk
Triaged
High
Geert JM Vanderkelen

Bug Description

The inner loop of send_plain (and send_compressed) looks like this:

while pktlen:
    pktlen -= self.sock.send(pkt)

On an incomplete self.sock.send, this will send the same data (from the start of pkt) on each loop iteration. Can I suggest something like this:

sent_ok = 0
while sent_ok < pktlen:
    sent_ok += self.sock.send(pkt[sent_ok:])

the key here is the use of "pkt[sent_ok:]" to advance the buffer pointer on each loop iteration.

I have encountered this problem when sending very large records.

Geert JM Vanderkelen (geertjmvdk)
Changed in myconnpy:
assignee: nobody → Geert JM Vanderkelen (geertjmvdk)
Revision history for this message
Geert JM Vanderkelen (geertjmvdk) wrote :

Sorry for not replying this to this bug earlier:

Can we have some test case or an error which is thrown when you see the problem? Or how big are your records: one big field? lots of fields, but rather big ones?

Changed in myconnpy:
status: New → Invalid
status: Invalid → Incomplete
Revision history for this message
David Creemer (dcreemer+launchpad) wrote :

Sorry for the incomplete information. I am using MySQL Connector/Python on top of gevent (gevent.org) which does ansync networking. My application connects to DB, and reads/writes to a table that has 10 columns, one of which is a LONG TEXT that can have values from 100 bytes to 100K bytes.

The key here is that the socket write (sock.send) is not guaranteed to send all of the data of course, only to return the count of bytes actually sent. If you are unfamiliar with gevent, you can try installing it, then doing this in some test code:

import gevent
import gevent.monkey
gevent.monkey.patch_all()

then run a test of large writes, and you should see a protocol error.

Revision history for this message
n1ywb (jeff-laughlin) wrote :

Geert, no offense, but this is a silly bug. Whoever wrote that code flunked sockets 101. It's not reasonable to ask the reporter to produce the test case for something so obvious. Just fix it.

You can switch to using socket.sendall(string[, flags]) which essentially implements the suggested loop.

See also http://docs.python.org/library/socket.html

See also http://docs.python.org/howto/sockets.html#using-a-socket which explains this issue.

Revision history for this message
David Creemer (dcreemer+launchpad) wrote :

Is development of this project ongoing? I apologize if it sounds rude -- I very much appreciate the work that's gone into the product. I'm just trying to decide if I should look into something else. Thanks!

Revision history for this message
Geert JM Vanderkelen (geertjmvdk) wrote :

@David: yes, just lots of stuff to do.. and priority goes to family after work.

@n1ywb: Not sure why I didn't use sendall() back then, but I'll look into it. Only happens with gevent? Odd?

Changed in myconnpy:
importance: Undecided → High
Revision history for this message
Geert JM Vanderkelen (geertjmvdk) wrote :

Please check the latest checkout of 0.3 to see if using using socket.sendall() fixes (or works around??) this bug.
 bzr branch lp:myconnpy/0.3

Changed in myconnpy:
status: Incomplete → In Progress
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.