[Linux-cluster] failover domain and service start
emmanuel segura
emi2fast at gmail.com
Thu Dec 19 09:05:18 UTC 2013
other thing, change the rgmanager loglevel, for more information you can
use this link https://fedorahosted.org/cluster/wiki/RGManager
2013/12/19 emmanuel segura <emi2fast at gmail.com>
> grep "/vms_c" /proc/mounts in every node
>
>
> 2013/12/18 Paras pradhan <pradhanparas at gmail.com>
>
>> I see this:
>>
>> Dec 18 16:17:18 cvtst3 clurgmgrd[13935]: <notice> Starting stopped
>> service vm:guest1
>> Dec 18 16:17:19 cvtst3 clurgmgrd[13935]: <notice> start on vm "guest1"
>> returned 1 (generic error)
>> Dec 18 16:17:19 cvtst3 clurgmgrd[13935]: <warning> #68: Failed to start
>> vm:guest1; return value: 1
>> Dec 18 16:17:19 cvtst3 clurgmgrd[13935]: <notice> Stopping service
>> vm:guest1
>>
>>
>>
>> On Wed, Dec 18, 2013 at 4:39 PM, emmanuel segura <emi2fast at gmail.com>wrote:
>>
>>> can you show the log?
>>>
>>>
>>> 2013/12/18 Paras pradhan <pradhanparas at gmail.com>
>>>
>>>> Added but same problem. vm does not start.
>>>>
>>>> -Paras.
>>>>
>>>>
>>>> On Wed, Dec 18, 2013 at 1:13 PM, emmanuel segura <emi2fast at gmail.com>wrote:
>>>>
>>>>> from script vm.sh i saw it try to discovery the hypervisor you are
>>>>> using, with hypervisor="xen" you force the script to use xen
>>>>>
>>>>>
>>>>> 2013/12/18 Paras pradhan <pradhanparas at gmail.com>
>>>>>
>>>>>> The only parameter I don't have is: hypervisor="xen"
>>>>>>
>>>>>> Does it matter?
>>>>>>
>>>>>> This is what i have:
>>>>>>
>>>>>> <vm autostart="1" domain="myfd1" exclusive="0" max_restarts="0"
>>>>>> name="guest1" path="/vms_c" recovery="restart" restart_expire_time="0"
>>>>>> use_virsh="0"/>
>>>>>>
>>>>>> -Paras.
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 18, 2013 at 12:24 PM, emmanuel segura <emi2fast at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> <vm name="guest1" hypervisor="xen" path="/vms_c" use_virsh="0">
>>>>>>>
>>>>>>> Incriment the config version of cluster.conf and ccs_tool update
>>>>>>> /etc/cluster/cluster.conf
>>>>>>>
>>>>>>>
>>>>>>> 2013/12/18 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>
>>>>>>>> Emmauel,
>>>>>>>>
>>>>>>>> With export OCF_RESKEY_name="guest1" ; export
>>>>>>>> OCF_RESKEY_path="/vms_c" ; export OCF_RESKEY_use_virsh=0
>>>>>>>>
>>>>>>>> , I can start the vm using : /usr/share/cluster/vm.sh
>>>>>>>>
>>>>>>>> I am wondering how to make the changes to cluser.conf or other
>>>>>>>> files so that we can start the vm using clucvcsadm.
>>>>>>>>
>>>>>>>> -Thanks
>>>>>>>> sorry for the delay.
>>>>>>>>
>>>>>>>> Paras.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Dec 5, 2013 at 12:36 PM, Paras pradhan <
>>>>>>>> pradhanparas at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Emmanue no.l I was busy on some other things I will test and let
>>>>>>>>> you know asap !
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Dec 5, 2013 at 12:26 PM, emmanuel segura <
>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Paras
>>>>>>>>>>
>>>>>>>>>> did you solved the problem?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Emmanuel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/11/25 emmanuel segura <emi2fast at gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Hello Paras
>>>>>>>>>>>
>>>>>>>>>>> Maybe i found the solution, in function validate_all we got
>>>>>>>>>>>
>>>>>>>>>>> if [ -z "$OCF_RESKEY_hypervisor" ] ||
>>>>>>>>>>> [ "$OCF_RESKEY_hypervisor" = "auto" ]; then
>>>>>>>>>>> export OCF_RESKEY_hypervisor="`virsh version |
>>>>>>>>>>> grep \"Running hypervisor:\" | awk '{print $3}' | tr A-Z a-z`"
>>>>>>>>>>> if [ -z "$OCF_RESKEY_hypervisor" ]; then
>>>>>>>>>>> ocf_log err "Could not determine
>>>>>>>>>>> Hypervisor"
>>>>>>>>>>> return $OCF_ERR_ARGS
>>>>>>>>>>> fi
>>>>>>>>>>> echo Hypervisor: $OCF_RESKEY_hypervisor
>>>>>>>>>>> fi
>>>>>>>>>>>
>>>>>>>>>>> #
>>>>>>>>>>> # Xen hypervisor only for when use_virsh = 0.
>>>>>>>>>>> #
>>>>>>>>>>> if [ "$OCF_RESKEY_use_virsh" = "0" ]; then
>>>>>>>>>>> if [ "$OCF_RESKEY_hypervisor" != "xen" ]; then
>>>>>>>>>>> ocf_log err "Cannot use
>>>>>>>>>>> $OCF_RESKEY_hypervisor hypervisor without using virsh"
>>>>>>>>>>> return $OCF_ERR_ARGS
>>>>>>>>>>> fi
>>>>>>>>>>>
>>>>>>>>>>> with this following enviroment variables, when i tested by hand
>>>>>>>>>>> the angent uses xm commands
>>>>>>>>>>>
>>>>>>>>>>> env | grep OCF
>>>>>>>>>>> OCF_RESKEY_hypervisor=xen
>>>>>>>>>>> OCF_RESKEY_path=/vms_c
>>>>>>>>>>> OCF_RESKEY_name=guest1
>>>>>>>>>>> OCF_RESKEY_use_virsh=0
>>>>>>>>>>>
>>>>>>>>>>> [root at client ~]# /usr/share/cluster/vm.sh status
>>>>>>>>>>> Management tool: xm
>>>>>>>>>>> <err> Cannot find 'xm'; is it installed?
>>>>>>>>>>> [vm.sh] Cannot find 'xm'; is it installed?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I don't have xen installed to test it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> if [ -n "$OCF_RESKEY_xmlfile" ]; then
>>>>>>>>>>> ocf_log err "Cannot use xmlfile if
>>>>>>>>>>> use_virsh is set to 0"
>>>>>>>>>>> return $OCF_ERR_ARGS
>>>>>>>>>>> fi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/11/25 emmanuel segura <emi2fast at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Hello paras
>>>>>>>>>>>>
>>>>>>>>>>>> missing the export command in front of variables, the correct
>>>>>>>>>>>> way is this
>>>>>>>>>>>>
>>>>>>>>>>>> export OCF_RESKEY_name="guest1" ; export
>>>>>>>>>>>> OCF_RESKEY_path="/vms_c" ; export OCF_RESKEY_use_virsh=0
>>>>>>>>>>>> [root at client ~]# env | grep OCF
>>>>>>>>>>>> OCF_RESKEY_path=/vms_c
>>>>>>>>>>>> OCF_RESKEY_name=guest1
>>>>>>>>>>>> OCF_RESKEY_use_virsh=0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/11/25 emmanuel segura <emi2fast at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Paras
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have a centos 6, i don't know if it is different on redhat
>>>>>>>>>>>>> 5, but i saw in the script vm.sh calls do_start function when start
>>>>>>>>>>>>> parameter is given
>>>>>>>>>>>>>
>>>>>>>>>>>>> do_start()
>>>>>>>>>>>>> {
>>>>>>>>>>>>> if [ "$OCF_RESKEY_use_virsh" = "1" ]; then
>>>>>>>>>>>>> do_virsh_start $*
>>>>>>>>>>>>> return $?
>>>>>>>>>>>>> fi
>>>>>>>>>>>>>
>>>>>>>>>>>>> do_xm_start $*
>>>>>>>>>>>>> return $?
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> i don't know why because the vm.sh uses virsh when you launch
>>>>>>>>>>>>> the script by hand :(
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/11/25 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looks like use_virsh=0 has no effect.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> [root at cvtst3 ~]# export OCF_RESKEY_name="guest1" ;
>>>>>>>>>>>>>> OCF_RESKEY_path="/vms_c" ; OCF_RESKEY_use_virsh=0
>>>>>>>>>>>>>> [root at cvtst3 ~]# set -x
>>>>>>>>>>>>>> ++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'
>>>>>>>>>>>>>> [root at cvtst3 ~]# /usr/share/cluster/vm.sh start
>>>>>>>>>>>>>> + /usr/share/cluster/vm.sh start
>>>>>>>>>>>>>> Hypervisor: xen
>>>>>>>>>>>>>> Management tool: virsh
>>>>>>>>>>>>>> Hypervisor URI: xen:///
>>>>>>>>>>>>>> Migration URI format: xenmigr://target_host/
>>>>>>>>>>>>>> Virtual machine guest1 is error: failed to get domain 'guest1'
>>>>>>>>>>>>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <debug> virsh -c xen:/// start guest1
>>>>>>>>>>>>>> error: failed to get domain 'guest1'
>>>>>>>>>>>>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'
>>>>>>>>>>>>>> [root at cvtst3 ~]# set +x
>>>>>>>>>>>>>> + set +x
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 22, 2013 at 5:22 PM, emmanuel segura <
>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hellos Paras
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Stop the vm and retry to start the vm with following
>>>>>>>>>>>>>>> commands and if you got some error show it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"; OCF_RESKEY_use_virsh=0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> set -x
>>>>>>>>>>>>>>> /usr/share/cluster/vm.sh start
>>>>>>>>>>>>>>> set +x
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I found the workaround to my issue. What i did is:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> run the vm using xm and then start using cluvscadm. This
>>>>>>>>>>>>>>>> works for me for the time being but I am not sure what is causing this.
>>>>>>>>>>>>>>>> This is what I did
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> xm create /vms_c/guest1
>>>>>>>>>>>>>>>> clusvcadm -e vm: guest1 ( This detects that guest1 is up
>>>>>>>>>>>>>>>> and quickly changes its status to success)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Although i used virt-install, it also create a xem format
>>>>>>>>>>>>>>>> configuration file and since use_virsh=0 it should be able to use this xen
>>>>>>>>>>>>>>>> format config file. This is my vm configuration:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> name = "guest1"
>>>>>>>>>>>>>>>> maxmem = 2048
>>>>>>>>>>>>>>>> memory = 512
>>>>>>>>>>>>>>>> vcpus = 1
>>>>>>>>>>>>>>>> #cpus="1-2"
>>>>>>>>>>>>>>>> bootloader = "/usr/bin/pygrub"
>>>>>>>>>>>>>>>> on_poweroff = "destroy"
>>>>>>>>>>>>>>>> on_reboot = "restart"
>>>>>>>>>>>>>>>> on_crash = "restart"
>>>>>>>>>>>>>>>> vfb = [ ]
>>>>>>>>>>>>>>>> disk = [ "tap:aio:/vms_c/guest1.img,xvda,w",
>>>>>>>>>>>>>>>> "tap:aio:/vms_c/guest1-disk.img,xvdb,w" ]
>>>>>>>>>>>>>>>> vif = [ "rate=10MB/s,mac=00:16:3e:6b:be:71,bridge=xenbr0" ]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for you help Emmanuel ! Really appreciate it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Nov 22, 2013 at 11:10 AM, emmanuel segura <
>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ok, but your vm doesn't start on others nodes, i think,
>>>>>>>>>>>>>>>>> for configuration problems
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ================================================================
>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> start on
>>>>>>>>>>>>>>>>> vm "guest1" returned 1 (generic error)
>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <warning> #68:
>>>>>>>>>>>>>>>>> Failed to start vm:guest1; return value: 1
>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> Stopping
>>>>>>>>>>>>>>>>> service vm:guest1
>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service
>>>>>>>>>>>>>>>>> vm:guest1 is recovering
>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <warning> #71:
>>>>>>>>>>>>>>>>> Relocating failed service vm:guest1
>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service
>>>>>>>>>>>>>>>>> vm:guest1 is stopped
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ================================================================
>>>>>>>>>>>>>>>>> in few words, try in every cluster node
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> set -x
>>>>>>>>>>>>>>>>> /usr/share/cluster/vm.sh start
>>>>>>>>>>>>>>>>> /usr/share/cluster/vm.sh stop
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> after you check if your vm can start and stop on every
>>>>>>>>>>>>>>>>> cluster node,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> /usr/share/cluster/vm.sh start
>>>>>>>>>>>>>>>>> /usr/share/cluster/vm.sh migrate name_of_a_cluster_node
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> can you show me your vm configuration under /vms_c?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> Emmanuel
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> And also to test I made use_virsh=1 , same problem. The
>>>>>>>>>>>>>>>>>> vm does not start up if the FD domains are offline.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Nov 22, 2013 at 10:37 AM, Paras pradhan <
>>>>>>>>>>>>>>>>>> pradhanparas at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Well thats seems to theoretically correct. But right now
>>>>>>>>>>>>>>>>>>> my cluser has use_virsh=0 and I don't have any issue untill my mebmers on
>>>>>>>>>>>>>>>>>>> the failover domains are offline. So wondering what is it that clusvcadm -e
>>>>>>>>>>>>>>>>>>> is looking when I don't use virsh .
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Nov 22, 2013 at 10:05 AM, emmanuel segura <
>>>>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If you used virt-install, i think you need use virsh,
>>>>>>>>>>>>>>>>>>>> the cluster uses xm xen command if you got use_virsh=0 and virsh if you got
>>>>>>>>>>>>>>>>>>>> use_virsh=1 in your cluster config
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I use virt-install to create virtual machines. Is
>>>>>>>>>>>>>>>>>>>>> there a way to debug why clusvcadm -e vm:guest1 is failing? vm.sh seems to
>>>>>>>>>>>>>>>>>>>>> use virsh and my cluster.conf has use_virsh=0
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Paras.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Nov 21, 2013 5:53 PM, "emmanuel segura" <
>>>>>>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> but did you configure your vm with xen tools or using
>>>>>>>>>>>>>>>>>>>>>> virt-manager?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Well no i don't want to use virsh. But as we are
>>>>>>>>>>>>>>>>>>>>>>> debugging with virsh now i found a strange issue.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I exported an xml file and imported to all nodes .
>>>>>>>>>>>>>>>>>>>>>>> Ran
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>> name="guest1" path="/vms_c"
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> export OCF_RESKEY_name="guest1" ;
>>>>>>>>>>>>>>>>>>>>>>> OCF_RESKEY_path="/vms_c"
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> set -x
>>>>>>>>>>>>>>>>>>>>>>> /usr/share/cluster/vm.sh start
>>>>>>>>>>>>>>>>>>>>>>> set +x
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> vm starts now. BUT from a cluster service : cluvscam
>>>>>>>>>>>>>>>>>>>>>>> -e vm:guest1 , same error.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> So if i populate all my domains' config files to all
>>>>>>>>>>>>>>>>>>>>>>> my cluser nodes and make use_virsh=1, then the issue is resolved. But this
>>>>>>>>>>>>>>>>>>>>>>> is a lot of work for those who have hundreds of vm.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> vm.start uses virsh . Is there a way to tell him not
>>>>>>>>>>>>>>>>>>>>>>> use virsh?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>> Paras.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 5:19 PM, emmanuel segura <
>>>>>>>>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> if you are using virsh for manage your vms, change
>>>>>>>>>>>>>>>>>>>>>>>> this in your cluster.conf
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>> use_virsh="0"
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> use_virsh="1"
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think i found the problem.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> virsh list --all does not show my vm . This is
>>>>>>>>>>>>>>>>>>>>>>>>> because it was created on another node. and another node has it. Now I want
>>>>>>>>>>>>>>>>>>>>>>>>> to start the service on a different node in which it was not created or
>>>>>>>>>>>>>>>>>>>>>>>>> where virsh list --all does not have an entry. Is it possible to create
>>>>>>>>>>>>>>>>>>>>>>>>> this entry using a xen config file?Looks like this is now a Xen issue
>>>>>>>>>>>>>>>>>>>>>>>>> rather than a linux-cluster issue . :)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Paras.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:58 PM, emmanuel segura <
>>>>>>>>>>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 1:did you verify your xen livemigration
>>>>>>>>>>>>>>>>>>>>>>>>>> configuration?
>>>>>>>>>>>>>>>>>>>>>>>>>> 2: where you vm disk reside?
>>>>>>>>>>>>>>>>>>>>>>>>>> 3: can you see your vm defined on every cluster
>>>>>>>>>>>>>>>>>>>>>>>>>> node with xm list?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> This is what I get
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hypervisor: xen
>>>>>>>>>>>>>>>>>>>>>>>>>>> Management tool: virsh
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hypervisor URI: xen:///
>>>>>>>>>>>>>>>>>>>>>>>>>>> Migration URI format: xenmigr://target_host/
>>>>>>>>>>>>>>>>>>>>>>>>>>> Virtual machine guest1 is error: failed to get
>>>>>>>>>>>>>>>>>>>>>>>>>>> domain 'guest1'
>>>>>>>>>>>>>>>>>>>>>>>>>>> error: Domain not found:
>>>>>>>>>>>>>>>>>>>>>>>>>>> xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> <debug> virsh -c xen:/// start guest1
>>>>>>>>>>>>>>>>>>>>>>>>>>> error: failed to get domain 'guest1'
>>>>>>>>>>>>>>>>>>>>>>>>>>> error: Domain not found:
>>>>>>>>>>>>>>>>>>>>>>>>>>> xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> ++ printf '\033]0;%s@%s:%s\007' root vtst3 '~'
>>>>>>>>>>>>>>>>>>>>>>>>>>> [root at cvtst3 ~]# set +x
>>>>>>>>>>>>>>>>>>>>>>>>>>> + set +x
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I am wondering why it failed to get domain .
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:43 PM, emmanuel segura
>>>>>>>>>>>>>>>>>>>>>>>>>>> <emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> yes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Well it is guest1. Isn't it?.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <vm autostart="1" domain="myfd1" exclusive="0"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> max_restarts="0" name="guest1" path="/vms_c" recovery="restart"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> restart_expire_time="0" use_virsh="0"/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It is a vm service if it matters.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:22 PM, emmanuel
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segura <emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use the servicename you defined in your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cluster.conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Says:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Running in test mode.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> No resource guest1 of type service found
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:07 PM, emmanuel
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segura <emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rg_test test /etc/cluster/cluster.conf
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start service guest1
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My failover domain looks like this:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <failoverdomain name="myfd1"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nofailback="1" ordered="1" restricted="0">
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <failoverdomainnode name="vtst1" priority="1"/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <failoverdomainnode name="vtst3" priority="2"/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <failoverdomainnode name="vtst2" priority="3"/>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> </failoverdomain>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have vm service that uses this failover
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> domain. If my node vtst1 is offline, the service doesnot start on vtst3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which is 2nd in the priority.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I tried to start it with: clusvcadm -e
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> vm:guest1 and even with -F and -m option.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> All i see is this error:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <notice> start on vm "guest1" returned 1 (generic error)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <warning> #68: Failed to start vm:guest1; return value: 1
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <notice> Stopping service vm:guest1
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <notice> Service vm:guest1 is recovering
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <warning> #71: Relocating failed service vm:guest1
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <notice> Service vm:guest1 is stopped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do I debug?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Paras.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quiera
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quiera
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios
>>>>>>>>>>>>>>>>>>>>>>>>>>>> quiera
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios
>>>>>>>>>>>>>>>>>>>>>>>>>> quiera
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Linux-cluster mailing list
>>>>>>>> Linux-cluster at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>
>>>>>>> --
>>>>>>> Linux-cluster mailing list
>>>>>>> Linux-cluster at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Linux-cluster mailing list
>>>>>> Linux-cluster at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>
>>>>> --
>>>>> Linux-cluster mailing list
>>>>> Linux-cluster at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>
>>>>
>>>>
>>>> --
>>>> Linux-cluster mailing list
>>>> Linux-cluster at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>
>>>
>>>
>>>
>>> --
>>> esta es mi vida e me la vivo hasta que dios quiera
>>>
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>
>>
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
>
> --
> esta es mi vida e me la vivo hasta que dios quiera
>
--
esta es mi vida e me la vivo hasta que dios quiera
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20131219/e7d460ad/attachment.htm>
More information about the Linux-cluster
mailing list