[Linux-cluster] failover domain and service start
emmanuel segura
emi2fast at gmail.com
Mon Nov 25 20:42:11 UTC 2013
Hello Paras
I have a centos 6, i don't know if it is different on redhat 5, but i saw
in the script vm.sh calls do_start function when start parameter is given
do_start()
{
if [ "$OCF_RESKEY_use_virsh" = "1" ]; then
do_virsh_start $*
return $?
fi
do_xm_start $*
return $?
}
i don't know why because the vm.sh uses virsh when you launch the script by
hand :(
2013/11/25 Paras pradhan <pradhanparas at gmail.com>
> Looks like use_virsh=0 has no effect.
>
> --
> [root at cvtst3 ~]# export OCF_RESKEY_name="guest1" ;
> OCF_RESKEY_path="/vms_c" ; OCF_RESKEY_use_virsh=0
> [root at cvtst3 ~]# set -x
> ++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'
> [root at cvtst3 ~]# /usr/share/cluster/vm.sh start
> + /usr/share/cluster/vm.sh start
> Hypervisor: xen
> Management tool: virsh
> Hypervisor URI: xen:///
> Migration URI format: xenmigr://target_host/
> Virtual machine guest1 is error: failed to get domain 'guest1'
> error: Domain not found: xenUnifiedDomainLookupByName
>
> <debug> virsh -c xen:/// start guest1
> error: failed to get domain 'guest1'
> error: Domain not found: xenUnifiedDomainLookupByName
>
> ++ printf '\033]0;%s@%s:%s\007' root cvtst3 '~'
> [root at cvtst3 ~]# set +x
> + set +x
> ---
>
>
> -Paras.
>
>
> On Fri, Nov 22, 2013 at 5:22 PM, emmanuel segura <emi2fast at gmail.com>wrote:
>
>> Hellos Paras
>>
>> Stop the vm and retry to start the vm with following commands and if you
>> got some error show it
>>
>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c" ;
>> OCF_RESKEY_use_virsh=0
>>
>>
>> set -x
>> /usr/share/cluster/vm.sh start
>> set +x
>>
>>
>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>
>>> I found the workaround to my issue. What i did is:
>>>
>>> run the vm using xm and then start using cluvscadm. This works for me
>>> for the time being but I am not sure what is causing this. This is what I
>>> did
>>>
>>> xm create /vms_c/guest1
>>> clusvcadm -e vm: guest1 ( This detects that guest1 is up and quickly
>>> changes its status to success)
>>>
>>> Although i used virt-install, it also create a xem format configuration
>>> file and since use_virsh=0 it should be able to use this xen format config
>>> file. This is my vm configuration:
>>>
>>> ---
>>> name = "guest1"
>>> maxmem = 2048
>>> memory = 512
>>> vcpus = 1
>>> #cpus="1-2"
>>> bootloader = "/usr/bin/pygrub"
>>> on_poweroff = "destroy"
>>> on_reboot = "restart"
>>> on_crash = "restart"
>>> vfb = [ ]
>>> disk = [ "tap:aio:/vms_c/guest1.img,xvda,w",
>>> "tap:aio:/vms_c/guest1-disk.img,xvdb,w" ]
>>> vif = [ "rate=10MB/s,mac=00:16:3e:6b:be:71,bridge=xenbr0" ]
>>>
>>> ---
>>>
>>> Thanks for you help Emmanuel ! Really appreciate it.
>>>
>>> -Paras.
>>>
>>>
>>> On Fri, Nov 22, 2013 at 11:10 AM, emmanuel segura <emi2fast at gmail.com>wrote:
>>>
>>>> ok, but your vm doesn't start on others nodes, i think, for
>>>> configuration problems
>>>> ================================================================
>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> start on vm "guest1"
>>>> returned 1 (generic error)
>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <warning> #68: Failed to start
>>>> vm:guest1; return value: 1
>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> Stopping service
>>>> vm:guest1
>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is
>>>> recovering
>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <warning> #71: Relocating
>>>> failed service vm:guest1
>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice> Service vm:guest1 is
>>>> stopped
>>>> ================================================================
>>>> in few words, try in every cluster node
>>>>
>>>>
>>>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"
>>>>
>>>> set -x
>>>> /usr/share/cluster/vm.sh start
>>>> /usr/share/cluster/vm.sh stop
>>>>
>>>> after you check if your vm can start and stop on every cluster node,
>>>>
>>>> /usr/share/cluster/vm.sh start
>>>> /usr/share/cluster/vm.sh migrate name_of_a_cluster_node
>>>>
>>>> can you show me your vm configuration under /vms_c?
>>>>
>>>> Thanks
>>>> Emmanuel
>>>>
>>>>
>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>
>>>>> And also to test I made use_virsh=1 , same problem. The vm does not
>>>>> start up if the FD domains are offline.
>>>>>
>>>>> -Paras.
>>>>>
>>>>>
>>>>> On Fri, Nov 22, 2013 at 10:37 AM, Paras pradhan <
>>>>> pradhanparas at gmail.com> wrote:
>>>>>
>>>>>> Well thats seems to theoretically correct. But right now my cluser
>>>>>> has use_virsh=0 and I don't have any issue untill my mebmers on the
>>>>>> failover domains are offline. So wondering what is it that clusvcadm -e is
>>>>>> looking when I don't use virsh .
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 22, 2013 at 10:05 AM, emmanuel segura <emi2fast at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> If you used virt-install, i think you need use virsh, the cluster
>>>>>>> uses xm xen command if you got use_virsh=0 and virsh if you got use_virsh=1
>>>>>>> in your cluster config
>>>>>>>
>>>>>>>
>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>
>>>>>>>> I use virt-install to create virtual machines. Is there a way to
>>>>>>>> debug why clusvcadm -e vm:guest1 is failing? vm.sh seems to use virsh and
>>>>>>>> my cluster.conf has use_virsh=0
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Paras.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Nov 21, 2013 5:53 PM, "emmanuel segura" <emi2fast at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> but did you configure your vm with xen tools or using virt-manager?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>
>>>>>>>>>> Well no i don't want to use virsh. But as we are debugging with
>>>>>>>>>> virsh now i found a strange issue.
>>>>>>>>>>
>>>>>>>>>> I exported an xml file and imported to all nodes . Ran
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> name="guest1" path="/vms_c"
>>>>>>>>>>
>>>>>>>>>> export OCF_RESKEY_name="guest1" ; OCF_RESKEY_path="/vms_c"
>>>>>>>>>>
>>>>>>>>>> set -x
>>>>>>>>>> /usr/share/cluster/vm.sh start
>>>>>>>>>> set +x
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> vm starts now. BUT from a cluster service : cluvscam -e vm:guest1
>>>>>>>>>> , same error.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So if i populate all my domains' config files to all my cluser
>>>>>>>>>> nodes and make use_virsh=1, then the issue is resolved. But this is a lot
>>>>>>>>>> of work for those who have hundreds of vm.
>>>>>>>>>>
>>>>>>>>>> vm.start uses virsh . Is there a way to tell him not use virsh?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Paras.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Nov 21, 2013 at 5:19 PM, emmanuel segura <
>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> if you are using virsh for manage your vms, change this in your
>>>>>>>>>>> cluster.conf
>>>>>>>>>>>
>>>>>>>>>>> from
>>>>>>>>>>> use_virsh="0"
>>>>>>>>>>> to
>>>>>>>>>>> use_virsh="1"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/11/22 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> I think i found the problem.
>>>>>>>>>>>>
>>>>>>>>>>>> virsh list --all does not show my vm . This is because it was
>>>>>>>>>>>> created on another node. and another node has it. Now I want to start the
>>>>>>>>>>>> service on a different node in which it was not created or where virsh list
>>>>>>>>>>>> --all does not have an entry. Is it possible to create this entry using a
>>>>>>>>>>>> xen config file?Looks like this is now a Xen issue rather than a
>>>>>>>>>>>> linux-cluster issue . :)
>>>>>>>>>>>>
>>>>>>>>>>>> Paras.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:58 PM, emmanuel segura <
>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 1:did you verify your xen livemigration configuration?
>>>>>>>>>>>>> 2: where you vm disk reside?
>>>>>>>>>>>>> 3: can you see your vm defined on every cluster node with xm
>>>>>>>>>>>>> list?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is what I get
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hypervisor: xen
>>>>>>>>>>>>>> Management tool: virsh
>>>>>>>>>>>>>> Hypervisor URI: xen:///
>>>>>>>>>>>>>> Migration URI format: xenmigr://target_host/
>>>>>>>>>>>>>> Virtual machine guest1 is error: failed to get domain 'guest1'
>>>>>>>>>>>>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <debug> virsh -c xen:/// start guest1
>>>>>>>>>>>>>> error: failed to get domain 'guest1'
>>>>>>>>>>>>>> error: Domain not found: xenUnifiedDomainLookupByName
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ++ printf '\033]0;%s@%s:%s\007' root vtst3 '~'
>>>>>>>>>>>>>> [root at cvtst3 ~]# set +x
>>>>>>>>>>>>>> + set +x
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am wondering why it failed to get domain .
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:43 PM, emmanuel segura <
>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> yes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Well it is guest1. Isn't it?.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <vm autostart="1" domain="myfd1" exclusive="0"
>>>>>>>>>>>>>>>> max_restarts="0" name="guest1" path="/vms_c" recovery="restart"
>>>>>>>>>>>>>>>> restart_expire_time="0" use_virsh="0"/>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It is a vm service if it matters.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:22 PM, emmanuel segura <
>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> use the servicename you defined in your cluster.conf
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Says:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Running in test mode.
>>>>>>>>>>>>>>>>>> No resource guest1 of type service found
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Paras.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Nov 21, 2013 at 4:07 PM, emmanuel segura <
>>>>>>>>>>>>>>>>>> emi2fast at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> rg_test test /etc/cluster/cluster.conf start service
>>>>>>>>>>>>>>>>>>> guest1
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/11/21 Paras pradhan <pradhanparas at gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> My failover domain looks like this:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> <failoverdomain name="myfd1" nofailback="1" ordered="1"
>>>>>>>>>>>>>>>>>>>> restricted="0">
>>>>>>>>>>>>>>>>>>>> <failoverdomainnode
>>>>>>>>>>>>>>>>>>>> name="vtst1" priority="1"/>
>>>>>>>>>>>>>>>>>>>> <failoverdomainnode
>>>>>>>>>>>>>>>>>>>> name="vtst3" priority="2"/>
>>>>>>>>>>>>>>>>>>>> <failoverdomainnode
>>>>>>>>>>>>>>>>>>>> name="vtst2" priority="3"/>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> </failoverdomain>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have vm service that uses this failover domain. If my
>>>>>>>>>>>>>>>>>>>> node vtst1 is offline, the service doesnot start on vtst3 which is 2nd in
>>>>>>>>>>>>>>>>>>>> the priority.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I tried to start it with: clusvcadm -e vm:guest1 and
>>>>>>>>>>>>>>>>>>>> even with -F and -m option.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> All i see is this error:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice> start
>>>>>>>>>>>>>>>>>>>> on vm "guest1" returned 1 (generic error)
>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <warning> #68:
>>>>>>>>>>>>>>>>>>>> Failed to start vm:guest1; return value: 1
>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:29 vtst3 clurgmgrd[13911]: <notice>
>>>>>>>>>>>>>>>>>>>> Stopping service vm:guest1
>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice>
>>>>>>>>>>>>>>>>>>>> Service vm:guest1 is recovering
>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <warning> #71:
>>>>>>>>>>>>>>>>>>>> Relocating failed service vm:guest1
>>>>>>>>>>>>>>>>>>>> Nov 21 15:40:35 vtst3 clurgmgrd[13911]: <notice>
>>>>>>>>>>>>>>>>>>>> Service vm:guest1 is stopped
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> How do I debug?
>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>> Paras.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Linux-cluster mailing list
>>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Linux-cluster mailing list
>>>>>>>>> Linux-cluster at redhat.com
>>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Linux-cluster mailing list
>>>>>>>> Linux-cluster at redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>>>>
>>>>>>> --
>>>>>>> Linux-cluster mailing list
>>>>>>> Linux-cluster at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Linux-cluster mailing list
>>>>> Linux-cluster at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> esta es mi vida e me la vivo hasta que dios quiera
>>>>
>>>> --
>>>> Linux-cluster mailing list
>>>> Linux-cluster at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>>
>>>
>>>
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>>
>>
>>
>>
>> --
>> esta es mi vida e me la vivo hasta que dios quiera
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
--
esta es mi vida e me la vivo hasta que dios quiera
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20131125/db229d21/attachment.htm>
More information about the Linux-cluster
mailing list