[libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa

Wed Jun 21 01:50:49 UTC 2017

Yep!
> That's actually a known bug that has been there "forever". It has always
> been brought up in the context of "I want communication between my NATed
> networks, and it only works in one direction!", rather than "I want to
> block communication between my NATed networks, but it gets past in one
> direction!". When people learn that it's (kind of) blocked, they work
> around it by adding a new network that is shared by all the guests.
> Because all of the reports have been in this direction, fixing it
> properly has been very low on (my) priority list.

Thanks for confirming the bug as I suspected, Laine! I knew it was too good
to be true that I'd be the first to discover this. Thanks for your huge
help, Martin.

I tried searching for this age-old you may be speaking of and I was only
able to find this:
https://bugzilla.redhat.com/show_bug.cgi?id=533193

Also found another somewhat-related bug affecting iptables (port-forwarding
to VM):
https://bugzilla.redhat.com/show_bug.cgi?id=1079088

If it's not too much trouble, can you provide me the ID number so I can
track it?

Aside from the extra packet processing overhead caused by the high
> number of rules needed (which may be completely insignificant), and the
> necessity to add/remove rules at specific labels rather than just
> inserting them all at the beginning, there is the issue that currently
> each iptables rule added for a libvirt network is dependent only on that
> network, so it's simple to determine which rules to remove when a
> network is destroyed. If we start adding rules that are dependent on two
> networks, then proper cleanup will be more complicated.

> None of that should prevent a properly motivated person from writing the
> code to do it, of course!

At least now I know this is quite feasible to fix. Thank you both for the
well-elaborated responses.

Cheers,

-Travis

On Tue, Jun 20, 2017 at 12:46 PM, Laine Stump <laine at laine.org> wrote:

> On 06/20/2017 05:27 AM, Martin Kletzander wrote:
> > On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander wrote:
> >> On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
> >>> Hello,
> >>>
> >>> I came across an interesting problem in my home lab a few weeks ago
> >>> as I'm
> >>> prepping for my RHCE exam using Michael Jang study guide. I've been
> >>> at this
> >>> for days now, and I still can't wrap my head around how two or more
> >>> virtual
> >>> networks in default NAT configuration are even allowed to communicate
> >>> with
> >>> each other despite what the libvirt documentation said.
> >>>
> >>>
> >>> Here's the excerpt I'm referring to in the wiki link here:
> >>> http://wiki.libvirt.org/page/Networking#Forwarding_
> Incoming_Connections:
> >>>
> >>>> By default, guests that are connected via a virtual network with
> >>>> <forward
> >>>> mode='nat'/> can make any outgoing network connection they like.
> >>>> Incoming
> >>>> connections are allowed from the host, and from other guests
> >>>> connected to
> >>>> the same libvirt network, but all other incoming connections are
> >>>> blocked by
> >>>> iptables rules.
> >>>
> >
> > I did not read this properly...
> >
> >>>
> >>> Also here's another assertion from 'The virtual network driver'
> >>> section in
> >>> http://libvirt.org/firewall.html:
> >>>
> >>>> type=nat
> >>>>
> >>>> Allow inbound related to an established connection. Allow outbound,
> but
> >>>> only from our expected subnet. Allow traffic between guests. Deny
> >>>> all other
> >
> > I would expect the 'traffic between guests' to mean only guests on the
> > same network.  Also that lines up with what's written above.
>
> That is correct.
>
> >
> >>>> inbound. Deny all other outbound.
> >>>
> >
> > [...]
> >
> >>
> >> Thanks for reporting this.  It's clearly a bug in libvirt.  The rules
> >> are in this order:
> >>
> >>  all rules for virbr0
> >>  all rules for virbr1
> >>  all rules for virbr2
> >>
> >> But what we should do instead is:
> >>
> >>  input rules for all networks
> >>  local rules for all networks
> >>  output rules for all networks
> >>  reject rules for all networks
>
> That doesn't achieve the desired results (as you figured out later,
> according to your followup comments.
>
>
> >>
> >> The problem is that we do not know how other rules look like.  So what
> >> we might need to do is create chains where rules for the first network
> >> are, then only append network rules into those chains.
> >>
> >> Would you mind filing a bug for this issue, so we can properly track it
> >> and don't forget about it?  I'll have a look at it in the meantime, but
> >> don't promise anything since I'm not that familiar with that part of the
> >> codebase.
> >>
> >
> > Actually, now that I'm thinking about it, the problem is not that we
> > disallow the communication from virbr1 to virbr0, but the problem is
> > that we allow the connection from virbr0 to everywhere.
>
> Yep!
>
> That's actually a known bug that has been there "forever". It has always
> been brought up in the context of "I want communication between my NATed
> networks, and it only works in one direction!", rather than "I want to
> block communication between my NATed networks, but it gets past in one
> direction!". When people learn that it's (kind of) blocked, they work
> around it by adding a new network that is shared by all the guests.
> Because all of the reports have been in this direction, fixing it
> properly has been very low on (my) priority list.
>
>
> >
> > Maybe the solution would be: for each network, insert the rules on top
> > of the forward chain and for each started network, explicitly reject
> > that one.  So that after one network starts it would look like this (I'm
> > writing this from memory just to illustrate the idea, not actually
> > looking up how stuff looks):
> >
> >  ACCEPT any virbr0 RELATED,ESTABLISHED
> >  ACCEPT virbr0 virbr0
> >  ACCEPT virbr0 any src:192.168.122.0/24
> >  REJECT any virbr0
> >  REJECT virbr0 any
> >
> > And after second network is started, we'd have:
> >
> >  # This one is new:
> >  REJECT virbr1 virbr0
>
> Yeah, it's (mostly) this rule that makes the solution complicated.
> Essentially you need one of these rules for each pair of virtual
> networks, so if you have 10 networks, you'll need 45 rules, for 11
> you'll need 55 (in general, for "n" networks, you'll need (n * (n-1) /
> 2) extra rules).
>
> Aside from the extra packet processing overhead caused by the high
> number of rules needed (which may be completely insignificant), and the
> necessity to add/remove rules at specific labels rather than just
> inserting them all at the beginning, there is the issue that currently
> each iptables rule added for a libvirt network is dependent only on that
> network, so it's simple to determine which rules to remove when a
> network is destroyed. If we start adding rules that are dependent on two
> networks, then proper cleanup will be more complicated.
>
> None of that should prevent a properly motivated person from writing the
> code to do it, of course!
>
> >  # These ones would be normally at the end, IIRC:
> >  ACCEPT any virbr1 RELATED,ESTABLISHED
> >  ACCEPT virbr1 virbr1
> >  ACCEPT virbr1 any src:192.168.122.0/24
> >  REJECT any virbr1
> >  REJECT virbr1 any
> >  # These are left as they were:
> >  ACCEPT any virbr0 RELATED,ESTABLISHED
> >  ACCEPT virbr0 virbr0
> >  ACCEPT virbr0 any src:192.168.122.0/24
> >  REJECT any virbr0
> >  REJECT virbr0 any
> >
> > Cc'ing Laine so that he can weigh in as he has way more knowledge of
> > this part of the code =)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20170620/690cea98/attachment.htm>