[libvirt-users] Networking issues with lxc containers in AWS EC2

[Laine Stump]

On 04/01/2016 07:04 PM, Peter Steele wrote:
> On 04/01/2016 02:07 PM, Laine Stump wrote:
>> On 03/31/2016 06:43 PM, Peter Steele wrote:
>>> I've created an EC2 AMI for AWS that essentially represents a CentOS 
>>> 7 "hypervisor" image. I deploy instances of these in AWS and create 
>>> an number of libvirt based lxc containers on each of these 
>>> instances. The containers run fine within a single host and have no 
>>> problem communicating with themselves as well as with their host, 
>>> and vice versa. However, containers hosted in one EC2 instance 
>>> cannot communicate with containers hosted in another EC2 instance.
>>>
>>> We've tried various tweaks with our Amazon VPC but have been unable 
>>> to find a way to solve this networking issue. If I use something 
>>> like VMware or KVM and create VMs using this same hypervisor image, 
>>> the containers running under these VMs can communicate with with 
>>> each other, even across different hosts.
>>
>> What is the <interface> config of your nested containers? Do they 
>> each get a public IP address?
> Yes, they all have public IPs on the same subnet. When deployed in a 
> VM environment on premises, the containers have no problems. Amazon 
> clearly does something with the packets though and the containers 
> can't talk to each other.


You say they can talk among containers on the same host, and with their 
own host (I guess you mean the virtual machine that is hosting the 
containers), but not to containers on another host. Can the containers 
communicate outside of the host at all? If not, perhaps the problem is 
iptables rules for the bridge device the containers are using - try 
running this command:

   sysctl net.bridge.bridge-nf-call-iptables

If that returns:

   net.bridge.bridge-nf-call-iptables = 1

then run this command and see if the containers can now communicate with 
the outside:

   sysctl -w net.bridge.bridge-nf-call-iptables=0


>>>
>>> My real question is has anyone tried deploying EC2 images that host 
>>> containers and have figured out how to successfully communicate 
>>> between containers on different hosts?
>>>
>>
>> No experience with EC2, sorry.
>>
> I think we'll need to go to Amazon themselves to resolve this issue. 
> There is very little information out there about how to get lxc 
> containers to work properly in EC2.

Well, if they've allowed your virtual machine to acquire multiple IP 
addresses, then it would make sense that they would allow them to 
actually use those IP addresses. I'm actually more inclined to think 
that the packets simply aren't getting out of the virtual machine (or 
the responses aren't getting back in).