[Linux-cluster] failover domain and service start

emmanuel segura emi2fast at gmail.com
Mon Nov 25 21:45:55 UTC 2013


Hello Paras

Maybe i found the solution, in function validate_all we got

       if [ -z "$OCF_RESKEY_hypervisor" ] ||
           [ "$OCF_RESKEY_hypervisor" = "auto" ]; then
                export OCF_RESKEY_hypervisor="`virsh version | grep
\"Running hypervisor:\" | awk '{print $3}' | tr A-Z a-z`"
                if [ -z "$OCF_RESKEY_hypervisor" ]; then
                        ocf_log err "Could not determine Hypervisor"
                        return $OCF_ERR_ARGS
                fi
                echo Hypervisor: $OCF_RESKEY_hypervisor
        fi

        #
        # Xen hypervisor only for when use_virsh = 0.
        #
        if [ "$OCF_RESKEY_use_virsh" = "0" ]; then
                if [ "$OCF_RESKEY_hypervisor" != "xen" ]; then
                        ocf_log err "Cannot use $OCF_RESKEY_hypervisor
hypervisor without using virsh"
                        return $OCF_ERR_ARGS
                fi

with this following enviroment variables, when i tested by hand the angent
uses xm commands

env | grep OCF
OCF_RESKEY_hypervisor=xen
OCF_RESKEY_path=/vms_c
OCF_RESKEY_name=guest1
OCF_RESKEY_use_virsh=0

[root at client ~]# /usr/share/cluster/vm.sh status
Management tool: xm
<err>    Cannot find 'xm'; is it installed?
[vm.sh] Cannot find 'xm'; is it installed?


I don't have xen installed to test it


                if [ -n "$OCF_RESKEY_xmlfile" ]; then
                        ocf_log err "Cannot use xmlfile if use_virsh is set
to 0"
                        return $OCF_ERR_ARGS
                fi



2013/11/25 emmanuel segura <emi2fast at gmail.com>

> Hello paras
>
> missing the export command in front of variables, the correct way is this
>
> export OCF_RESKEY_name="guest1" ; export OCF_RESKEY_path="/vms_c" ; export
> OCF_RESKEY_use_virsh=0
> [root at client ~]# env | grep OCF
> OCF_RESKEY_path=/vms_c
> OCF_RESKEY_name=guest1
> OCF_RESKEY_use_virsh=0
>
>
>
> 2013/11/25 emmanuel segura <emi2fast at gmail.com>
>
>> Hello Paras
>>
>> I have  a centos 6, i don't know if it is different on redhat 5, but i
>> saw in the script vm.sh calls do_start function when start parameter is
>> given
>>
>> do_start()
>> {
>>         if [ "$OCF_RESKEY_use_virsh" = "1" ]; then
>>                 do_virsh_start $*
>>                 return $?
>>         fi
>>
>>         do_xm_start $*
>>         return $?
>> }
>>
>> i don't know why because the vm.sh uses virsh when you launch the script
>> by hand :(
>>
>>
>> 2013/11/25 Paras pradhan <pradhanparas at gmail.com>
>>
>>> Looks like use_virsh=0 has no effect.
>>>
>>> --
>>> [root at cvtst3 ~]# export OCF_RESKEY_name="guest1" ;
>>> OCF_RESKEY_path="/vms_c" ; OCF_RESKEY_use_virsh=0
>>> [root at cvtst3 ~]# set -x
>>> ++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'
>>> [root at cvtst3 ~]# /usr/share/cluster/vm.sh start
>>> + /usr/share/cluster/vm.sh start
>>> Hypervisor: xen
>>> Management tool: virsh
>>> Hypervisor URI: xen:///
>>> Migration URI format: xenmigr://target_host/
>>> Virtual machine guest1 is error: failed to get domain 'guest1'
>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>
>>> <debug>  virsh -c xen:/// start guest1
>>> error: failed to get domain 'guest1'
>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>
>>> ++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'
>>>  [root at cvtst3 ~]# set +x
>>> + set +x
>>> ---
>>>
>>>
>>> -Paras.
>>>
>>>
>>> On Fri, Nov 22, 2013 at 5:22 PM, emmanuel segura <emi2fast at gmail.com>wrote:
>>>
>>>> Hellos Paras
>>>>
>>>> Stop the vm and retry to start the vm with following commands and if
>>>> you got some error show it
>>>>
>>>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c" ;
>>>> OCF_RESKEY_use_virsh=0
>>>>
>>>>
>>>> set -x
>>>> /usr/share/cluster/vm.sh start
>>>> set +x
>>>>
>>>>
>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>
>>>>> I found the workaround to my issue. What i did is:
>>>>>
>>>>> run the vm using xm and then start using cluvscadm. This works for me
>>>>> for the time being but I am not sure what is causing this. This is what I
>>>>> did
>>>>>
>>>>> xm create /vms_c/guest1
>>>>> clusvcadm -e vm: guest1 ( This detects that guest1 is up and quickly
>>>>> changes its status to success)
>>>>>
>>>>> Although i used virt-install, it also create a xem format
>>>>> configuration file and since use_virsh=0 it should be able to use this xen
>>>>> format config file. This is my vm configuration:
>>>>>
>>>>> ---
>>>>> name = "guest1"
>>>>> maxmem = 2048
>>>>> memory = 512
>>>>> vcpus = 1
>>>>> #cpus="1-2"
>>>>> bootloader = "/usr/bin/pygrub"
>>>>> on_poweroff = "destroy"
>>>>> on_reboot = "restart"
>>>>> on_crash = "restart"
>>>>> vfb = [  ]
>>>>> disk = [ "tap:aio:/vms_c/guest1.img,xvda,w",
>>>>> "tap:aio:/vms_c/guest1-disk.img,xvdb,w" ]
>>>>> vif = [ "rate=10MB/s,mac=00:16:3e:6b:be:71,bridge=xenbr0" ]
>>>>>
>>>>> ---
>>>>>
>>>>> Thanks for you help Emmanuel ! Really appreciate it.
>>>>>
>>>>> -Paras.
>>>>>
>>>>>
>>>>> On Fri, Nov 22, 2013 at 11:10 AM, emmanuel segura <emi2fast at gmail.com>wrote:
>>>>>
>>>>>> ok, but your vm doesn't start on others nodes, i think, for
>>>>>> configuration problems
>>>>>> ================================================================
>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> start on vm "guest1"
>>>>>> returned 1 (generic error)
>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <warning> #68: Failed to
>>>>>> start vm:guest1; return value: 1
>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> Stopping service
>>>>>> vm:guest1
>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is
>>>>>> recovering
>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <warning> #71: Relocating
>>>>>> failed service vm:guest1
>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is
>>>>>> stopped
>>>>>> ================================================================
>>>>>> in few words, try in every cluster node
>>>>>>
>>>>>>
>>>>>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"
>>>>>>
>>>>>> set -x
>>>>>> /usr/share/cluster/vm.sh start
>>>>>> /usr/share/cluster/vm.sh stop
>>>>>>
>>>>>> after you check if your vm can start and stop on every cluster node,
>>>>>>
>>>>>> /usr/share/cluster/vm.sh start
>>>>>> /usr/share/cluster/vm.sh migrate name_of_a_cluster_node
>>>>>>
>>>>>> can you show me your vm configuration under /vms_c?
>>>>>>
>>>>>> Thanks
>>>>>> Emmanuel
>>>>>>
>>>>>>
>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>
>>>>>>> And also to test I made use_virsh=1 , same problem. The vm does not
>>>>>>> start up if the FD domains are offline.
>>>>>>>
>>>>>>> -Paras.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Nov 22, 2013 at 10:37 AM, Paras pradhan <
>>>>>>> pradhanparas at gmail.com> wrote:
>>>>>>>
>>>>>>>> Well thats seems to theoretically correct. But right now my cluser
>>>>>>>> has use_virsh=0 and I don't have any issue untill my mebmers on the
>>>>>>>> failover domains are offline. So wondering what is it that clusvcadm -e is
>>>>>>>> looking when I don't use virsh .
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 22, 2013 at 10:05 AM, emmanuel segura <
>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> If you used virt-install, i think you need use virsh, the cluster
>>>>>>>>> uses xm xen command if you got use_virsh=0 and virsh if you got use_virsh=1
>>>>>>>>> in your cluster config
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>
>>>>>>>>>> I use virt-install to create virtual machines. Is there a way to
>>>>>>>>>> debug why clusvcadm -e vm:guest1 is failing? vm.sh  seems to use virsh and
>>>>>>>>>> my cluster.conf has use_virsh=0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Paras.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Nov 21, 2013 5:53 PM, "emmanuel segura" <emi2fast at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> but did you configure your vm with xen tools or using
>>>>>>>>>>> virt-manager?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Well no i don't want to use virsh.   But as we are debugging
>>>>>>>>>>>> with virsh now i found a strange issue.
>>>>>>>>>>>>
>>>>>>>>>>>> I exported an xml file and imported to all nodes . Ran
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> name="guest1" path="/vms_c"
>>>>>>>>>>>>
>>>>>>>>>>>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"
>>>>>>>>>>>>
>>>>>>>>>>>> set -x
>>>>>>>>>>>> /usr/share/cluster/vm.sh start
>>>>>>>>>>>> set +x
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> vm starts now. BUT from a cluster service : cluvscam -e
>>>>>>>>>>>> vm:guest1 , same error.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So if i populate all my domains' config files to all my cluser
>>>>>>>>>>>> nodes and make use_virsh=1, then the issue is resolved. But this is a lot
>>>>>>>>>>>> of work for those who have hundreds of vm.
>>>>>>>>>>>>
>>>>>>>>>>>> vm.start uses virsh . Is there a way to tell him not use virsh?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Paras.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 21, 2013 at 5:19 PM, emmanuel segura <
>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> if you are using virsh for manage your vms, change this in
>>>>>>>>>>>>> your cluster.conf
>>>>>>>>>>>>>
>>>>>>>>>>>>> from
>>>>>>>>>>>>> use_virsh="0"
>>>>>>>>>>>>> to
>>>>>>>>>>>>> use_virsh="1"
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think i found the problem.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> virsh list --all does not show my vm . This is because it was
>>>>>>>>>>>>>> created on another node. and another node has it. Now I want to start the
>>>>>>>>>>>>>> service on a different node in which it was not created or where virsh list
>>>>>>>>>>>>>> --all does not have an entry. Is it possible to create this entry using a
>>>>>>>>>>>>>> xen config file?Looks like this is now a Xen issue rather than a
>>>>>>>>>>>>>> linux-cluster issue . :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Paras.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:58 PM, emmanuel segura <
>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1:did you verify your xen livemigration configuration?
>>>>>>>>>>>>>>> 2: where you vm disk reside?
>>>>>>>>>>>>>>> 3: can you see your vm defined on every cluster node with xm
>>>>>>>>>>>>>>> list?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is what I get
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hypervisor: xen
>>>>>>>>>>>>>>>> Management tool: virsh
>>>>>>>>>>>>>>>> Hypervisor URI: xen:///
>>>>>>>>>>>>>>>> Migration URI format: xenmigr://target_host/
>>>>>>>>>>>>>>>> Virtual machine guest1 is error: failed to get domain
>>>>>>>>>>>>>>>> 'guest1'
>>>>>>>>>>>>>>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <debug>  virsh -c xen:/// start guest1
>>>>>>>>>>>>>>>> error: failed to get domain 'guest1'
>>>>>>>>>>>>>>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ++ printf '\033]0;%s@%s:%s\007' root vtst3 '~'
>>>>>>>>>>>>>>>> [root at cvtst3 ~]# set +x
>>>>>>>>>>>>>>>> + set +x
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  I am wondering why it failed to get domain .
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:43 PM, emmanuel segura <
>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> yes
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Well it is guest1. Isn't it?.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> <vm autostart="1" domain="myfd1" exclusive="0"
>>>>>>>>>>>>>>>>>> max_restarts="0" name="guest1" path="/vms_c" recovery="restart"
>>>>>>>>>>>>>>>>>> restart_expire_time="0" use_virsh="0"/>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It is a vm service if it matters.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:22 PM, emmanuel segura <
>>>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> use the servicename you defined in your cluster.conf
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Says:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Running in test mode.
>>>>>>>>>>>>>>>>>>>> No resource guest1 of type service found
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:07 PM, emmanuel segura <
>>>>>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> rg_test test /etc/cluster/cluster.conf start service
>>>>>>>>>>>>>>>>>>>>> guest1
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> My failover domain looks like this:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> <failoverdomain name="myfd1" nofailback="1"
>>>>>>>>>>>>>>>>>>>>>> ordered="1" restricted="0">
>>>>>>>>>>>>>>>>>>>>>>                                 <failoverdomainnode
>>>>>>>>>>>>>>>>>>>>>> name="vtst1" priority="1"/>
>>>>>>>>>>>>>>>>>>>>>>                                 <failoverdomainnode
>>>>>>>>>>>>>>>>>>>>>> name="vtst3" priority="2"/>
>>>>>>>>>>>>>>>>>>>>>>                                 <failoverdomainnode
>>>>>>>>>>>>>>>>>>>>>> name="vtst2" priority="3"/>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>                         </failoverdomain>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I have vm service that uses this failover domain. If
>>>>>>>>>>>>>>>>>>>>>> my node vtst1 is offline, the service doesnot start on vtst3 which is 2nd
>>>>>>>>>>>>>>>>>>>>>> in the priority.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I tried to start it with: clusvcadm -e vm:guest1
>>>>>>>>>>>>>>>>>>>>>> and even with -F and -m option.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> All i see is this error:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice>
>>>>>>>>>>>>>>>>>>>>>> start on vm "guest1" returned 1 (generic error)
>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <warning>
>>>>>>>>>>>>>>>>>>>>>> #68: Failed to start vm:guest1; return value: 1
>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice>
>>>>>>>>>>>>>>>>>>>>>> Stopping service vm:guest1
>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice>
>>>>>>>>>>>>>>>>>>>>>> Service vm:guest1 is recovering
>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <warning>
>>>>>>>>>>>>>>>>>>>>>> #71: Relocating failed service vm:guest1
>>>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice>
>>>>>>>>>>>>>>>>>>>>>> Service vm:guest1 is stopped
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> How do I debug?
>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>> Paras.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Linux-cluster mailing list
>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Linux-cluster mailing list
>>>>>>> Linux-cluster at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>
>>>>>> --
>>>>>> Linux-cluster mailing list
>>>>>> Linux-cluster at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Linux-cluster mailing list
>>>>> Linux-cluster at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>
>>>> --
>>>> Linux-cluster mailing list
>>>> Linux-cluster at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>
>>>
>>>
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>
>>
>>
>>
>> --
>> esta es mi vida e me la vivo hasta que dios quiera
>>
>
>
>
> --
> esta es mi vida e me la vivo hasta que dios quiera
>



-- 
esta es mi vida e me la vivo hasta que dios quiera
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20131125/cc6d8d98/attachment.htm>


More information about the Linux-cluster mailing list