`ssh-add -x` does not prevent key use

I add my key:

$ ssh-add -t $((60*60*2)) /media/secret/secret/id_dsa

After which I physically remove the USB stick with that key and I lock my agent:

$ ssh-add -x
Enter lock password:
Again:
Agent locked.

Then I ssh someplace I shouldn't be able to get to:

$ ssh -v 10.13.37.6
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 10.13.37.6 [10.13.37.6] port 22.
debug1: Connection established.
debug1: identity file /home/martin/.ssh/id_rsa type -1
debug1: identity file /home/martin/.ssh/id_rsa-cert type -1
debug1: identity file /home/martin/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/martin/.ssh/id_dsa-cert type -1
debug1: identity file /home/martin/.ssh/id_ecdsa type -1
debug1: identity file /home/martin/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1 Debian-4ubuntu5
debug1: match: OpenSSH_5.5p1 Debian-4ubuntu5 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA b9:b0:f8:f0:6c:36:5b:b1:55:7f:d3:e3:b6:4c:32:5d
debug1: Host '10.13.37.6' is known and matches the RSA host key.
debug1: Found key in /home/martin/.ssh/known_hosts:30
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /home/martin/.ssh/id_dsa
debug1: Server accepts key: pkalg ssh-dss blen 435
debug1: Authentication succeeded (publickey).
Authenticated to 10.13.37.6 ([10.13.37.6]:22).
debug1: channel 0: new [client-session]
debug1: Requesting <email address hidden>
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Linux bh-live-02 2.6.35-28-virtual #50-Ubuntu SMP Fri Mar 18 19:16:26 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.10

Notably, it says, "debug1: Offering DSA public key: /home/martin/.ssh/id_dsa", but:

$ ls -l /home/martin/.ssh/id_dsa
ls: cannot access /home/martin/.ssh/id_dsa: No such file or directory

Also, -l shows a key when it should show none:

$ ssh-add -l
1024 a5:51:2f:98:70:97:97:bb:54:0f:12:cf:72:f7:be:64 /media/secret/secret/id_dsa (DSA)

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: openssh-client 1:5.8p1-7ubuntu1
Uname: Linux 3.3.0-030300rc7-generic x86_64
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
Date: Mon Mar 26 17:52:37 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111011)
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
RelatedPackageVersions:
 ssh-askpass N/A
 libpam-ssh N/A
 keychain N/A
 ssh-askpass-gnome 1:5.8p1-7ubuntu1
SSHClientVersion: OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
SourcePackage: openssh
UpgradeStatus: No upgrade log present (probably fresh install)