snowball adb over usb doesn't work consistently

Looking at https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc46-igloo-stable-blob-12.07-release/#build=1 we are getting inconsistent results with adb over usb on snowball. For some, it works right away. For others, we need to go through several cycles of attach, reattach, restarting adb before finally seeing it connect. The exact combination and order of events to make it work reliably isn't known yet, but I was eventually able to get a connection to adb on my snowball.

However, for both me, and Vishal was seeing this also, we can reliably reproduce a problem where it fails to reattach once it's disconnected.

steps:
1. get adb over usb working
2. disconnect the usb cable
3. reconnect the usb cable
At this point, adb does not detect the device, and dmesg shows errors on usb. So far, I've been unable to make it work again without rebooting the snowball board.

Here's the dmesg from the snowball for the detach/reattach:
<6>[ 1777.668060] adb_release
<6>[ 1777.673431] android_work: sent uevent USB_STATE=DISCONNECTED
<6>[ 1777.728607] adb_open
<6>[ 1777.730865] rndis_function_bind_config MAC: 8E:1B:EC:58:84:A2
<4>[ 1777.736694] gadget: using random self ethernet address
<4>[ 1777.742279] gadget: using random host ethernet address
<6>[ 1777.752441] rndis0: MAC a6:58:03:43:37:78
<6>[ 1777.756469] rndis0: HOST MAC ca:db:ff:d9:0f:48
<6>[ 1777.762756] adb_bind_config
<6>[ 1805.840576] android_work: sent uevent USB_STATE=CONNECTED
<6>[ 1805.861236] android_work: sent uevent USB_STATE=DISCONNECTED
<6>[ 1805.932434] gadget: high-speed config #1: android
<6>[ 1805.937561] adb_release
<6>[ 1805.947174] android_work: sent uevent USB_STATE=CONNECTED
<6>[ 1806.008331] adb_open
<6>[ 1806.008361] rndis_function_bind_config MAC: CA:DB:FF:D9:0F:48
<4>[ 1806.008453] gadget: using random self ethernet address
<4>[ 1806.008453] gadget: using random host ethernet address
<6>[ 1806.013824] rndis0: MAC c6:12:00:7d:d5:1b
<6>[ 1806.013824] rndis0: HOST MAC 1a:8a:5a:ac:fc:90
<6>[ 1806.013854] adb_bind_config
<6>[ 1806.281860] android_work: sent uevent USB_STATE=DISCONNECTED
<6>[ 1806.352874] android_work: sent uevent USB_STATE=CONNECTED
<6>[ 1806.370727] android_work: sent uevent USB_STATE=DISCONNECTED
<6>[ 1806.444061] gadget: high-speed config #1: android
<6>[ 1806.445739] android_work: sent uevent USB_STATE=CONNECTED
<6>[ 1806.464752] android_work: sent uevent USB_STATE=CONFIGURED

And here's the dmesg from the host:
[612973.148639] usb 1-1.2: USB disconnect, device number 32
[613001.183677] usb 1-1.2: new high-speed USB device number 33 using ehci_hcd
[613001.283007] usb 1-1.2: can't set config #1, error -32
[613001.494779] usb 1-1.2: USB disconnect, device number 33
[613001.694552] usb 1-1.2: new high-speed USB device number 34 using ehci_hcd
[613006.773858] usb 1-1.2: can't set config #1, error -110