With the backport modules, it was even worse, one minute after connecting, the connection stopped working. My syslog is full of these message:

Apr 23 19:16:29 gwaihir kernel: [  437.202564] wlan0: authenticated
Apr 23 19:16:29 gwaihir kernel: [  437.202566] wlan0: associate with AP ffff88006ce09ac0
Apr 23 19:16:29 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  scanning -> associated 
Apr 23 19:16:29 gwaihir kernel: [  437.205916] wlan0: RX ReassocResp from ffff8800511e8016 (capab=0x431 status=0 aid=1)
Apr 23 19:16:29 gwaihir kernel: [  437.205919] wlan0: associated
Apr 23 19:16:32 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  associated -> disconnected 
Apr 23 19:16:32 gwaihir kernel: [  440.198034] wlan0: deauthenticated (Reason: 2)
Apr 23 19:16:32 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  disconnected -> scanning 
Apr 23 19:16:33 gwaihir kernel: [  441.196055] wlan0: direct probe to AP ffff88006ce09ac0 try 1
Apr 23 19:16:33 gwaihir kernel: [  441.396104] wlan0: direct probe to AP ffff88006ce09ac0 try 2
Apr 23 19:16:33 gwaihir kernel: [  441.399905] wlan0 direct probe responded
Apr 23 19:16:33 gwaihir kernel: [  441.399908] wlan0: authenticate with AP ffff88006ce09ac0
Apr 23 19:16:33 gwaihir kernel: [  441.402689] wlan0: authenticated
Apr 23 19:16:33 gwaihir kernel: [  441.402691] wlan0: associate with AP ffff88006ce09ac0
Apr 23 19:16:33 gwaihir kernel: [  441.600109] wlan0: associate with AP ffff88006ce09ac0
Apr 23 19:16:33 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  scanning -> associated 
Apr 23 19:16:33 gwaihir kernel: [  441.605087] wlan0: RX ReassocResp from ffff88004d8f0016 (capab=0x431 status=0 aid=1)
Apr 23 19:16:33 gwaihir kernel: [  441.605091] wlan0: associated
Apr 23 19:16:36 gwaihir kernel: [  444.595897] wlan0: deauthenticated (Reason: 2)
Apr 23 19:16:36 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  associated -> disconnected 
Apr 23 19:16:36 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  disconnected -> scanning 
Apr 23 19:16:37 gwaihir kernel: [  445.592125] wlan0: direct probe to AP ffff88006ce09ac0 try 1
Apr 23 19:16:37 gwaihir kernel: [  445.595913] wlan0 direct probe responded
Apr 23 19:16:37 gwaihir kernel: [  445.595916] wlan0: authenticate with AP ffff88006ce09ac0
Apr 23 19:16:38 gwaihir kernel: [  445.792127] wlan0: authenticate with AP ffff88006ce09ac0
Apr 23 19:16:38 gwaihir kernel: [  445.794931] wlan0: authenticated
Apr 23 19:16:38 gwaihir kernel: [  445.794933] wlan0: associate with AP ffff88006ce09ac0
Apr 23 19:16:38 gwaihir kernel: [  445.799818] wlan0: RX ReassocResp from ffff88004d950016 (capab=0x431 status=0 aid=1)
Apr 23 19:16:38 gwaihir kernel: [  445.799821] wlan0: associated
Apr 23 19:16:38 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  scanning -> associated 
Apr 23 19:16:39 gwaihir NetworkManager: <info>  wlan0: link timed out. 
Apr 23 19:16:41 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  associated -> disconnected 
Apr 23 19:16:41 gwaihir kernel: [  448.794110] wlan0: deauthenticated (Reason: 2)
Apr 23 19:16:41 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  disconnected -> scanning 
Apr 23 19:16:42 gwaihir kernel: [  449.792109] wlan0: direct probe to AP ffff88006ce09ac0 try 1
Apr 23 19:16:42 gwaihir kernel: [  449.992105] wlan0: direct probe to AP ffff88006ce09ac0 try 2
Apr 23 19:16:42 gwaihir kernel: [  449.996078] wlan0 direct probe responded
Apr 23 19:16:42 gwaihir kernel: [  449.996080] wlan0: authenticate with AP ffff88006ce09ac0
Apr 23 19:16:42 gwaihir kernel: [  449.998821] wlan0: authenticated
Apr 23 19:16:42 gwaihir kernel: [  449.998824] wlan0: associate with AP ffff88006ce09ac0
Apr 23 19:16:42 gwaihir kernel: [  450.196102] wlan0: associate with AP ffff88006ce09ac0
Apr 23 19:16:42 gwaihir kernel: [  450.201065] wlan0: RX ReassocResp from ffff880051148016 (capab=0x431 status=0 aid=1)
Apr 23 19:16:42 gwaihir kernel: [  450.201068] wlan0: associated
Apr 23 19:16:42 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  scanning -> associated 
Apr 23 19:16:45 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  associated -> disconnected 
Apr 23 19:16:45 gwaihir kernel: [  453.191431] wlan0: deauthenticated (Reason: 2)
Apr 23 19:16:45 gwaihir NetworkManager: <info>  (wlan0): supplicant connection state:  disconnected -> scanning 
Apr 23 19:16:46 gwaihir kernel: [  454.188107] wlan0: direct probe to AP ffff88006ce09ac0 try 1

I purged the backport module packages for now. An unstable WLAN is better than none :)