[libvirt-users] Networking issues with lxc containers in AWS EC2

[Peter Steele]

On 04/11/2016 11:33 AM, Laine Stump wrote:
> Interesting. That functionality was moved out of the kernel's bridge 
> module into br_netfilter some time back, but that was done later than 
> the kernel 3.10 that is used by CentOS 7. Are you running some later 
> kernel version?
>
> If your kernel doesn't have a message in dmesg that looks like this:
>
>    bridge: automatic filtering via arp/ip/ip6tables has been deprecated.
>    Update your scripts to load br_netfilter if you need this.
>
> and the bridge driver is loaded, then that key should be available. Of 
> course if you don't have it, that's equivalent to having it set to 0, 
> so you should be okay regardless of why it's missing.
>
Ah, you were right. I'd forgot that the AMI I've using was one running 
the 4.0.5 ml kernel. We discovered that bonded interfaces running with 
mode 5 or 6 do not work with lxc containers (the host's ARP table does 
not get updated). The issue was fixed in the 4.0.5 kernel so we ran for 
a short time with this kernel, only to later abandon this kernel due to 
a bug with software RAID.

I've reverted the kernel back to 3.10 on the AWS instances I'm using the 
net.bridge.bridge-nf-call-iptables key is now present. It's already set 
to 0 though so there is nothing that needs to be done here.
>
> I wouldn't be too quick to judgement. First take a look at tcpdump on 
> the bridge interface that the containers are attached to, and on the 
> ethernet device that connects the bridge to the rest of Amazon's 
> infrastructure. If you see packets from the container's IP going out 
> but not coming back in, check the iptables rules (again - firewalld 
> uses iptables to setup its filtering) for a REJECT or DISCARD rule 
> that has an incrementing count. I use something like this to narrow 
> down the list I need to check:
>
> while true; do iptables -v -S -Z | grep -v '^Zeroing' | grep -v "c 0 
> 0" | grep -e '-c'; echo '**************'; sleep 1;
>
> If you don't see any REJECT or DISCARD rules being triggered, then 
> maybe the problem is that AWS is providing an IP address to your 
> container's MAC, but isn't actually allowing traffic from that MAC out 
> onto the network.
>
I'll get this test setup. Unfortunately I'm not particularly 
knowledgeable with iptables; we don't use it in our product so I've 
never had to deal with it. I think you are right though about what's 
happening--AWS doesn't recognize the MAC addresses for containers 
running under another instance.