[Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md

Dan Sneddon dsneddon at redhat.com
Fri Nov 13 21:10:26 UTC 2015

On 11/13/2015 12:56 PM, Dan Sneddon wrote:
> Hi Boris,
> Let's keep this on-list, there may be others who are having similar
> issues who could find this discussion useful.
> Answers inline...
> On 11/13/2015 12:17 PM, Boris Derzhavets wrote:
>> ________________________________________
>> From: Dan Sneddon <dsneddon at redhat.com>
>> Sent: Friday, November 13, 2015 2:46 PM
>> To: Boris Derzhavets; Javier Pena
>> Cc: rdo-list at redhat.com
>> Subject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
>> On 11/13/2015 11:38 AM, Boris Derzhavets wrote:
>>> I understand that in usual situation , creating ifcfg-br-ex and ifcfg-eth2 ( as OVS bridge and OVS port) ,
>>>  `service network restart` should be run to make eth2 (no IP) OVS port of br-ex (any IP which belongs ext net and is available)
>>> What bad does NetworkManager when external network provider is used ?
>>> Disabling it,  I break routing via eth0's interfaces of cluster nodes to ( ext net),
>>> so nothing is supposed to work :-
>>>    http://blog.oddbit.com/2014/05/28/multiple-external-networks-wit/
>>>    http://dbaxps.blogspot.com/2015/10/multiple-external-networks-with-single.html
>>> Either I am missing something here.
>>> ________________________________________
>>> From: rdo-list-bounces at redhat.com <rdo-list-bounces at redhat.com> on behalf of Boris Derzhavets <bderzhavets at hotmail.com>
>>> Sent: Friday, November 13, 2015 1:09 PM
>>> To: Javier Pena
>>> Cc: rdo-list at redhat.com
>>> Subject: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md
>>> Working on this task I was able to build 3 node HAProxy/Keepalived  Controller's cluster , create compute node , launch CirrOS VM,
>>> However, I cannot ping floating IP of VM running on compute ( total 4 CentOS 7.1 VMs, nested kvm enabled )
>>> Looks like provider external networks  doesn't work for me.
>>> But , to have eth0 without IP (due to `ovs-vsctl add-port br-eth0 eth0 ) still allowing to ping,
>>> I need NetworkManager active, rather then network.service
>>> [root at hacontroller1 network-scripts]# systemctl status NetworkManager
>>> NetworkManager.service - Network Manager
>>>    Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled)
>>>    Active: active (running) since Fri 2015-11-13 20:39:21 MSK; 12min ago
>>>  Main PID: 808 (NetworkManager)
>>>    CGroup: /system.slice/NetworkManager.service
>>>            ├─ 808 /usr/sbin/NetworkManager --no-daemon
>>>            └─2325 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0...
>>> Nov 13 20:39:22 hacontroller1.example.com NetworkManager[808]: <info>  NetworkManager state is n...L
>>> Nov 13 20:39:22 hacontroller1.example.com dhclient[2325]: bound to -- renewal in 1...s.
>>> Nov 13 20:39:22 hacontroller1.example.com NetworkManager[808]: <info>  (eth0): Activation: succe....
>>> Nov 13 20:39:25 hacontroller1.example.com NetworkManager[808]: <info>  startup complete
>>> [root at hacontroller1 network-scripts]# systemctl status network.service
>>> network.service - LSB: Bring up/down networking
>>>    Loaded: loaded (/etc/rc.d/init.d/network)
>>>    Active: inactive (dead)
>>> [root at hacontroller1 network-scripts]# cat ifcfg-eth0
>>> TYPE="Ethernet"
>>> BOOTPROTO="static"
>>> NAME="eth0"
>>> DEVICE=eth0
>>> ONBOOT="yes"
>>> [root at hacontroller1 network-scripts]# ping -c 3
>>> PING ( 56(84) bytes of data.
>>> 64 bytes from icmp_seq=1 ttl=64 time=0.087 ms
>>> 64 bytes from icmp_seq=2 ttl=64 time=0.128 ms
>>> 64 bytes from icmp_seq=3 ttl=64 time=0.117 ms
>>> --- ping statistics ---
>>> 3 packets transmitted, 3 received, 0% packet loss, time 1999ms
>>> rtt min/avg/max/mdev = 0.087/0.110/0.128/0.021 ms
>>> If I disable NetworkManager and enable network this feature will be lost. Eth0 would have to have static IP or dhcp lease,
>>> to provide route to
>>> Thank you.
>>> Boris.
>>> _______________________________________________
>>> Rdo-list mailing list
>>> Rdo-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>> OK, a few things here. First of all, you don't actually need to have an
>> IP address on the host system to use a VLAN or interface as an external
>> provider network. The Neutron router will have an IP on the right
>> network, and within its namespace will be able to reach the 10.10.10.x
>> network.
>>> It looks to me like NetworkManager is running dhclient for eth0, even
>>> though you have BOOTPROTO="static". This is causing an IP address to be
>>> added to eth0, so you are able to ping 10.10.10.x from the host. When
>>> you turn off NetworkManager, this unexpected behavior goes away, *but
>>> you should still be able to use provider networks*.
>>      Here I am quoting Lars Kellogg Stedman
>>            http://blog.oddbit.com/2014/05/28/multiple-external-networks-wit/
>>      The bottom statement in blog post above states :-
>>      "This assumes that eth1 is connected to a network using and eth2 is connected to a network using, and that each network has a gateway sitting at the corresponding .1 address."
> Right, what Lars means is that eth1 is physically connected to a
> network with the subnet, and eth2 is physically connected
> to a network with the subnet.
> You might notice that in Lars's instructions, he never puts a host IP
> on either interface.
>>> Try creating a Neutron router with an IP on 10.10.10.x, and then you
>>> should be able to ping that network from the router namespace.
>>    " When I issue `neutron router-creater --ha True --tenant-id xxxxxx RouterHA`  , i cannot specify router's
>>     IP " 
> Let me refer you to this page, which explains the basics of creating
> and managing Neutron networks:
> http://docs.openstack.org/user-guide/cli_create_and_manage_networks.html
> You will have to create an external network, which you will associate
> with a physical network via a bridge mapping. The default bridge
> mapping for br-ex is datacentre:br-ex.
> Using the name of the physical network "datacentre", we can create an
> external network:
> [If the external network is on VLAN 104]
> neutron net-create ext-net --router:external \
> --provider:physical_network datacentre \
> --provider:network_type vlan \
> --provider:segmentation_id 104
> [If the external net is on the native VLAN (flat)]
> neutron net-create ext-net --router:external \
> --provider:physical_network datacentre \
> --provider:network_type flat
> Next, you must create a subnet for the network, including the range of
> floating IPs (allocation pool):
> neutron subnet-create --name ext-subnet \
> --enable_dhcp=False \
> --allocation-pool start=,end= \
> --gateway \
> ext-net
> Next, you have to create a router:
> neutron router-create ext-router
> You then add an interface to the router. Since Neutron will assign the
> first address in the subnet to the router by default (, you
> will want to first create a port with a specific IP, then assign that
> port to the router.
> neutron port-create ext-net --fixed-ip ip_address=
> You will need to note the UUID of the newly created port. You can also
> see this with "neutron port-list". Now, create the router interface
> with the port you just created:
> neutron router-interface-add ext-router port=<UUID>
>>> If you want to be able to ping 10.10.10.x from the host, then you
>>> should put either a static IP or DHCP on the bridge, not on eth0. This
>>> should work whether you are running NetworkManager or network.service.
>>    "I do can ping 10.0.0.x from F23 KVM Server (running cluster's VMs as Controller's nodes), 
>>      it's  just usual non-default libvirt subnet,matching exactly external network creating in Javier's  "Howto".
>>      It was  created via `virsh net-define openstackvms.xml`, but I cannot ping FIPs belong to
>>      cloud VM on this subnet."
> I think you will have better luck once you create the external network
> and router. You can then use namespaces to ping the network from the
> router:
> First, obtain the qrouter-<UUID> from the list of namespaces:
> sudo ip netns list
> Then, find the qrouter-<UUID> and ping from there:
> ip netns exec qrouter-XXXX-XXXX-XXX-XXX ping

One more quick thing to note:

In order to use floating IPs, you will also have to attach the external
router to the tenant networks where floating IPs will be used.

When you go through the steps to create a tenant network, also attach
it to the router:

1) Create the network:

neutron net-create tenant-net-1

2) Create the subnet:

neutron subnet-create --name tenant-subnet-1 tenant-net-1

3) Attach the external router to the network:

neutron router-interface-add tenant-router-1 subnet=tenant-subnet-1

(since no specific port was given in the router-interface-add command,
Neutron will automatically choose the first address in the given
subnet, so in this example)

Dan Sneddon         |  Principal OpenStack Engineer
dsneddon at redhat.com |  redhat.com/openstack
650.254.4025        |  dsneddon:irc   @dxs:twitter

More information about the rdo-list mailing list