Unable to reach instances from their public IP address

On Ubuntu Maverick (Eucalyptus 2.0) I'm not able to reach the eucalyptus instances, due to the firewall rules. I didn't find exactly the problem, but I only know that it's iptables which drops packages.

Our setup is, a server with CC, Walrus and SC and two additional servers with NC, all servers have two network cards, one connected to our public LAN and another one connected to an isolated switch.

CC and walrus listen on the public LAN network, the SC and NC listen on the private LAN network.

We are able to launch instances and to connect EBS volumes without problems. From within the instances, we are able to connect to Internet without problems, either. However our problem comes when we try to connect to the instances using the public LAN IP address we assigned on installation time, all packages are dropped.

For the iptables rules I'm going to attach, we have the public IP address 10.82.3.1 assigned to the CC public interface (br0), which points to the 172.19.1.2 ip address assigned to the eucalyputs instance. I just opened the ping port:

sysadmin@europe:~$ sudo iptables -n -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- 172.19.0.0/16 169.254.169.254 tcp dpt:80 to:169.254.169.254:8773
DNAT all -- 0.0.0.0/0 10.82.3.1 to:172.19.1.2

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DNAT all -- 0.0.0.0/0 10.82.3.1 to:172.19.1.2

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- 172.19.1.2 !172.19.0.0/16 to:10.82.3.1
MASQUERADE all -- 172.19.0.0/16 !172.19.0.0/16
sysadmin@europe:~$ sudo iptables -n -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate ESTABLISHED
ACCEPT all -- 0.0.0.0/0 !172.19.0.0/16
build-build all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 172.19.1.0/27 172.19.1.0/27
LOG all -- 0.0.0.0/0 0.0.0.0/0 limit: avg 5/min burst 5 LOG flags 0 level 7 prefix `iptables denied (input): '

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain build-build (1 references)
target prot opt source destination
ACCEPT icmp -- 0.0.0.0 172.19.1.0/27

sysadmin@europe:~$ sudo iptables -n -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- 172.19.0.0/16 169.254.169.254 tcp dpt:80 to:169.254.169.254:8773
DNAT all -- 0.0.0.0/0 10.82.3.1 to:172.19.1.2

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DNAT all -- 0.0.0.0/0 10.82.3.1 to:172.19.1.2

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- 172.19.1.2 !172.19.0.0/16 to:10.82.3.1
MASQUERADE all -- 172.19.0.0/16 !172.19.0.0/16

And the configured network interfaces:

sysadmin@europe:~$ ifconfig
br0 Link encap:Ethernet HWaddr XXXXXXXXXXXX
          inet addr:10.82.0.10 Bcast:10.82.3.255 Mask:255.255.252.0
          inet6 addr: fe80::222:19ff:fe55:abd1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3154360 errors:0 dropped:0 overruns:0 frame:0
          TX packets:252607 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:250658946 (250.6 MB) TX bytes:555159076 (555.1 MB)

br1 Link encap:Ethernet HWaddr XXXXXXXXXXXXXXX
          inet addr:192.168.0.10 Bcast:192.168.0.255 Mask:255.255.255.0
          inet6 addr: fe80::222:19ff:fe55:abd3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2727761 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3336571 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1876704895 (1.8 GB) TX bytes:1622792007 (1.6 GB)

br0:pub Link encap:Ethernet HWaddr XXXXXXXXXX
          inet addr:10.82.3.1 Bcast:0.0.0.0 Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

br1:metadata Link encap:Ethernet HWaddr 00:22:19:55:ab:d3
          inet addr:169.254.169.254 Bcast:0.0.0.0 Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

br1:priv Link encap:Ethernet HWaddr XXXXXXXXXXX
          inet addr:172.19.1.1 Bcast:172.19.1.31 Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth0 Link encap:Ethernet HWaddr XXXXXXXXXXXX
          inet6 addr: fe80::222:19ff:fe55:abd1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:14477583 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11162887 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5865981327 (5.8 GB) TX bytes:7994928399 (7.9 GB)
          Interrupt:37 Memory:ec000000-ec012800

eth1 Link encap:Ethernet HWaddr XXXXXXXXXXXX
          inet6 addr: fe80::222:19ff:fe55:abd3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3407602 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3870025 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1973588029 (1.9 GB) TX bytes:1672350633 (1.6 GB)
          Interrupt:37 Memory:ea000000-ea012800

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:49909118 errors:0 dropped:0 overruns:0 frame:0
          TX packets:49909118 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8333123720 (8.3 GB) TX bytes:8333123720 (8.3 GB)

The server where we run the CC, walrus and SC is also used to execute regular KVM servers (nothing related with NC).

We are only able to ssh into the eucalyptus instances from within the CC using its 172.19.1.2 ip address.

As you can see on the previous iptables output, I added some logging rules to check where the problem is and this is the output I get while pinging from my laptop (10.82.5.1):

Apr 11 10:26:32 europe kernel: [466684.316651] iptables denied (input): IN=br0 OUT=br1 PHYSIN=eth0 SRC=10.82.5.1 DST=172.19.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=7014 SEQ=1428
Apr 11 10:26:44 europe kernel: [466696.402197] iptables denied (input): IN=br0 OUT=br1 PHYSIN=eth0 SRC=10.82.5.1 DST=172.19.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=7014 SEQ=1440
Apr 11 10:26:56 europe kernel: [466708.271690] iptables denied (input): IN=br0 OUT=br1 PHYSIN=eth0 SRC=10.82.5.1 DST=172.19.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=7014 SEQ=1452
Apr 11 10:27:08 europe kernel: [466720.242250] iptables denied (input): IN=br0 OUT=br1 PHYSIN=eth0 SRC=10.82.5.1 DST=172.19.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=7014 SEQ=1464

The only change I did manually to iptables was the addition of the logging rules, nothing more.

This used to be a 1.6 installation with Lucid, but I cleared everything while upgrading to Eucalyptus 2.0 / Maverick so from the Eucalyptus point of view, is a fresh installation.