[Linux-cluster] help on configuring a shared gfs volume in a load balanced http cluster
Alex
linux at vfemail.net
Fri Jul 25 15:37:54 UTC 2008
Hi Gordan,
Thanks for your reply... I have performed some small steps forward...
> > Actually, i want just to have hdb1 from shd1 and hdc1 from shd2 joided in
> > one volume. No mirror for this volume at that momment. Is possible? If
> > yes, how? Using ATAoE?
>
> Set up ATAoE on shd and use it to export a volume. Connect to this ATAoE
> share from the front end nodes. You can then use something like Cluster
> LVM (CLVM) to unify them into one volume.
Now, I setup ATAoE inside shd1 and shd2 and now, i am able to see exported
disks on my redhat cluster manager machine (rhclm), so should be the same
view on all our rs1, rs2 and rs3 webservers.
See below:
[root at rhclm ~]# cat /proc/partitions
major minor #blocks name
3 0 19551168 hda
3 1 98248 hda1
3 2 1 hda2
3 5 1000408 hda5
3 6 8673808 hda6
152 560 78150744 etherd/e2.3
152 561 39075088 etherd/e2.3p1
152 562 39075624 etherd/e2.3p2
152 288 39082680 etherd/e1.2
152 289 19541056 etherd/e1.2p1
152 290 19541592 etherd/e1.2p2
[root at rhclm ~]# aoe-stat
e1.2 40.020GB eth0 up
e2.3 80.026GB eth0 up
[root at rhclm ~]#
[root at rhclm ~]# ls -l /dev/etherd/*
brw-r----- 1 root disk 152, 288 Jul 25 17:47 /dev/etherd/e1.2
brw-r----- 1 root disk 152, 289 Jul 25 17:47 /dev/etherd/e1.2p1
brw-r----- 1 root disk 152, 290 Jul 25 17:47 /dev/etherd/e1.2p2
brw-r----- 1 root disk 152, 560 Jul 25 17:47 /dev/etherd/e2.3
brw-r----- 1 root disk 152, 561 Jul 25 17:47 /dev/etherd/e2.3p1
brw-r----- 1 root disk 152, 562 Jul 25 17:47 /dev/etherd/e2.3p2
[root at rhclm ~]#
So, is exacly what i exported from shd1 and shd2 servers (hdb -> 2*20GB slices
and hdc -> 2*40GB slices)
>From here i am lost. I understand from you, that now, on one of our
realservers, let say rs1, shoud i use CLVM to unify them. I want to obtain 2
volumes:
VOL1=e1.2p1+e2.3p1
VOL2=e1.2p2+e2.3p2
I am not sure how to do it using clvm... All is crossing my mind is mdadm...
mdadm -C /dev/md0 -l 0 -n 2 \
/dev/etherd/e1.2p1 /dev/etherd/e2.3p1
and after that, to use:
pvcreate /dev/md0
vgcreate extendible_lvm /dev/md0
lvcreate --extents 60GB --name extendible_lvm www_docroot
and finally mkfs.gfs /dev/extendible_lvm/www_docroot
Can you help me regading clvm?
Regards,
Alx
>
> Then create GFS on this volume.
>
> Note that if you lose either of the two shd machines you will likely lose
> all the data.
>
> > After that, i would like to know, how to install GFS on this volume and
> > use it as documennt root on our real web servers (rs1, rs2, rs3). Is
> > possible? If yes, how?
>
> Yes, when you have the logical volume consisting of shd1 and shd2, create
> the GFS on it as per the docs (mkfs.gfs), mount it to where you want it,
> and point Apache at that path. Nothing magical about it, it's just like
> any other once you have it mounted.
>
> > I don't understand from your explanation, how to group machines: shd1 and
> > shd2 should be in one cluster and rs1, rs2 and rs3 in other cluster
>
> I don't see why you need shd1 and shd2 machines in a cluster. They are
> just SANs. Unless they are mirroring each other or beign each other's
> backup there is no immediately obvious reason from your example why they
> should be clustered together.
>
> > or: shd1 and shd2 shoud be regular servers which is just exporting their
> > HDD
>
> Yes. And you don't export the HDD per se using ATAoE or iSCSI - you export
> a "volume" (which is just a file on shd's file system that is effectively
> a disk image).
>
> > using ATAoE and rs1, rs2 and rs2 to be grouped in one cluster which
> > are importing a GFS volume from somwhere?
>
> rs machines would import the ATAoE volumes, establish a logical volume on
> top of them, and then start the GFS file system on top of that.
>
> > If yes, from where? How can i configure a GFS volume on
> > ATAoE disks and from where will be accesible?
>
> It will be accessible from any machine in the cluster the GFS volume is
> built for (in this case rs set), once they connect the ATAoE (or iSCSI if
> that's what you use for it, there isn't THAT much difference between
> them) shares from shds.
>
> > I need another one machine
> > which will act as agregator for ATAoE disks or our real web servers (rs1,
> > rs2, rs3) will responsible to import directly these disks?
>
> You don't need an agregator, you can unify the volumes using CLVM into one
> big logical volume, and have GFS live on top of that.
>
> >> and ATAoE or iSCSI to export the
> >> volumes for the rs machines to mount.
> >
> > In our lab we are using regular hard disks, so iSCSI is excluded.
>
> iSCSI is a network protocol, nothing to do with SCSI disks per se.
> It's SCSI-over-ethernet. You can export any file on a machine as a volume
> using iSCSI. Whether the underlying disk is SCSI, ATA or something exotic
> is entirely irrelevant.
>
> ATAoE and iSCSI are both applicable to your case. ATAoE has somewhat lower
> overheads (read: a little faster) but is ethernet layer based. iSCSI is
> TCP based so is routable. iSCSI is also a little more mature.
>
> > I read an article here (http://www.linuxjournal.com/article/8149) about
> > ATAoE and i have some questions:
> >
> > - on our centos 5.2 boxes, we already have aoe kernel module but we don't
> > have aoe-stat command. Is any packet shoud i install via yum to have this
> > command (or other command to hadle aoe disks) or is required do download
> > aoetools-26.tar.gz and compile from source
> > (http://sourceforge.net/projects/aoetools/)
> >
> > - in above article they are talking about RAID10, LVM and JFS. They are
> > not teaching me about GFS and clustering. They choose JFS and not GFS
> > saying that "JFS is a filesystem that can grow dynamically to large
> > sizes, so he is going to put a JFS filesystem on a logical volume". I
> > want that but using GFS, is possible or not?
>
> There are several concepts and technologies you need to go read up on
> before getting further with this:
> ATAoE
> iSCSI
> LVM/CLVM for volume management
>
> If you add additional volumes (e.g. exported via iSCSI or ATAoE) to your
> SAN boxes, you can add them into your CLVM volume you have GFS on top of,
> and the virtual "disk" (logical volume) will show as being bigger. You can
> then grow the GFS file system on this volume and have it extend onto the
> additional space.
>
> > They are saying that:
> >
> > "using a cluster filesystem such as GFS, it is possible for multiple
> > hosts on the Ethernet network to access the same block storage using ATA
> > over Ethernet. There's no need for anything like an NFS server"
>
> NFS and GFS are sort of equivalent, layer wise.
>
> > "But there's a snag. Any time you're using a lot of disks, you're
> > increasing the chances that one of the disks will fail. Usually you use
> > RAID to take care of this issue by introducing some redundancy.
> > Unfortunately, Linux software RAID is not cluster-aware. That means each
> > host on the network cannot do RAID 10 using mdadm and have things simply
> > work out."
>
> What they are saying is that you can't export two ATAoE/iSCSI shares, have
> mdadm RAID on top, and then have GFS on top, because the mdadm layer isn't
> cluster aware. But you aren't using RAID on that level.
>
> RAID would be on the shd machines (hardware or mdadm RAID on the disks you
> use for storage, before any exporting via ATAoE or iSCSI happens.
>
> If you want the servers mirrored (i.e. RAID1), that's what you would use
> DRBD as I mentioned earlier. But then you wouldn't mount a share from each
> machine, you'd mount just one of the two, and have shds clustered for
> fail-over.
>
> > So, finally, what should i do? Can you or anybody suggest me some howtos
> > and what is the correct order to group machines and implement clustering?
>
> See above. Have a Google around for the things I mentioned, and ask more
> specific questions. :)
>
> Gordan
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
More information about the Linux-cluster
mailing list