[Linux-cluster] slow I/O on v7000 SAN

[Michael Bubb]

On 11/16/2011 11:54 AM, Steven Whitehouse wrote:
> Hi,
> 
> On Wed, 2011-11-16 at 11:42 -0500, Michael Bubb wrote:
>> Hello -
>>
>> We are experiencing extreme I/O slowness on a gfs2 volume on a SAN.
>>
>> We have a:
>>
>> Netezza TF24
>> IBM V7000 SAN
>> IBM Bladecenter with 3 HS22 blades
>> Stand alone HP DL380 G7 server
>>
>>
>> The 3 blades and the HP DL380 are clustered using RHEL 6.1 and
>> clustersuite 5.5.
>>
> You should ask the Red Hat support team about this, as they should be
> able to help.

I assumed that this would be assumed. I have had a ticket in for a
while. The ticket is not going anywhere. I have sent sosrepts, etc... I
also have aticket in with IBM regarding the SAN. After a day they came
back and told me "Yes, you have I/O slownesss...."

So I thought I would hit the community and see if this rings a bell with
anyone else.
> 
>> We have 2 clustered volumes on different storage pools (one has 10k
>> drives the other 7.2k).
>>
>> We have an internal test that reads a large file (950G) using fopen and
>> memmap. On a standalone server in a datacenter (Ubuntu Raid 5 10k disks)
>> the tests take approximately 75seconds to run.
>>
>> On the blades the test takes 300 - 350 seconds.
>>
>> I have been looking at the cluster conf; any gfs tuning I can find. I am
>> not really sure what I should post here?
>>
>> yrs
>>
>> Michael
>>
> So it is a streaming data test. Are you running it on all three nodes at
> the same time or just one when you get the 300 seconds times? Did you
> mount with noatime,nodiratime set?

We tested first all nodes, then one node. There is amazing consistency
here. He have run this test about 10 times in different scenarios and it
is always about 5 times slower than the Ubuntu SATA Raid5 volume.

> 
> Are the drives you are using just a straight linear lvm volume from a
> JBOD effectively?
THey are RAID6 mdisks on the SAN organized into storage pools. THe
volumes are created on top of this.
> 
> Steve.
>  
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster

-- 


Michael Bubb
System Administrator
..................................................
T: +1.646.380-2738  | C: +1.646.783.8769
mbubb at collective.com  | michael.bubb at gmail.com
..................................................
Collective | collective.com | Twitter: @collectivesays
99 Park Ave. 5th Floor |  New York, NY 10016 | +1.888.460.9513
..................................................

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20111116/dba7982b/attachment.sig>