Fedora SMP dual core, dual AMD 64 processor system

[Bryan J. Smith]

On Wed, 2005-08-24 at 09:57 -0400, Mark Hahn wrote:
> good god, that kind of condescension only comes from *knowing* you're wrong!
> I, like Bill, have never seen HW raid come anywhere close to the performance
> of SW raid on the same hardware.  yes, raid5, yes raid6, yes, even during rebuilds.
> (though why you'd be worried about performance during rebuilds is beyond me - 
> lots of failures?)

Then assume I'm wrong.

You guys must have been still using i960 instead of StrongARM or, more
recently, X-Scale over the last 5 years.

> and HW raid is flawless, I suppose.

_Select_ hardware RAID has been, with exceptions.

Same deal with LVM/MD, it's worked for some things, with exceptions.

> jeez, it's all software raid - 

The different is that I'm not pushing the data streams through my CPU-
memory interconnect.  That ties up important I/O.  Heck, it's why you
don't see your CPU states busy because nothing can be feed to it.  The
interconnect is saturated.

> you just prefer to trust software that's closed, difficult to upgrade,
> written by unknown persons and not auditable.

I trust specific vendors.

Adaptec?  Hell no.  DPT before Adaptec?  No, i960.
Intel?  Been a little late on using their own products (sadly enough).

Mylex?  Yes.  LSI Logic?  Yes.  3Ware?  Yes.

> I prefer to run open, auditable, trivially fixable raid that happens to 
> scale very nicely with host speed.

Then you obviously haven't used 3Ware.
Don't throw all vendors in the same boat.
Furthermore, don't assume there isn't auditing.

Again, I think you're oversimplifying many hardware solutions.

I've used MD, LVM and, more recently but only to a limited extent, LVM2.
I often use LVM/LVM2 to RAID-0 stripe multiple hardware RAID volumes
over multiple card/busses.  I like LVM, don't get me wrong.

And I _do_ use LVM when I don't have a good, intelligent, high-
performing hardware RAID solution.  That includes i960 designs that
moronic Tier-1 vendors like Dell still ship.

But when you have a 500MHz XScale with an internal interconnect that
ripples through I/O far better than a 3GHz CPU, and you aren't making
several redundant copies of the data in your host system as a result,
then it's faster and more efficient.

Same deal for MIPS, ARM and other ASIC designs where you're doing
RAID-0, 1, 10/1e, etc...  In fact, the new SAS controllers I've been
reading up on give you RAID-0, 1, 10/1e for free because it's so easy to
do in an ASIC (and cuts your transfers over the interconnect by 2x in
the case of RAID-1).

> non sequitur, since Bill didn't claim anything about 3dm's features.

I'm just saying that there _are_ lots of capabilities you guys haven't
used.  I've used _both_ MD/LVM _and_ newer storage designs.

> the ironic thing about spending $500 on a so-called HW raid card is that 
> you've *not* spent $400 on more disks.  really, there's no reason not to 
> go further and do HW raid with SCSI disks to really minimize your TB.

Honestly, with all the changes in MD/LVM I've seen over the years, 3Ware
and Mylex/LSI Logic have a far better track record with me.

> you must be running on PII's or something.  even my entry-level servers have 
> plenty of memory and bus bandwidth to handle SW raid5.

Then why don't you use them as Ethernet switches, routers, etc... too?
I think you're missing the point.

The purpose of your non-I/O interconnect is not to be replicating data
operations.  It is to be servicing requests for that data.

When you start taxing your non-I/O interconnect with added data streams,
that takes away it's ability to do other things.

> but really, why are 
> you worried about this?  it's a frickin fileserver!  you're worried it's
> going to impinge on your seti at home ranking?

Dude, you didn't understand the first thing I said.

I said that added RAID-4/5/6 overhead severely cuts into how much data
your fileserver can flop around.  Instead of the non-I/O interconnect
working on requested data, it is now replicating basic I/O operations as
well.

You can't get that kind of information from CPU states in Linux.  Heck,
even Linux's iostat is kinda immature.  That's because Linux is just
getting into not only the processor affinity of memory, but also I/O, in
the Opteron.

I've often found Solaris to be much better at tracking such interconnect
overhead and usage, and it is sizable when you're going from _0_% non-
I/O interconnect to doubling or even tripling what the normal service
streams are using.

This is fundamental system design gentlemen.

> how embarassing for you to so automatically assume ignorance.

Fine by me.

> because it's more cost-effective in either high-end or high-volume products.

Give me a break.  Not at 5+ disks.  Not at all.  Especially when I'm
turning the system architecture into a data pipe for _non_ user
services.

> plonk.

Again, fine by me.


-- 
Bryan J. Smith     b.j.smith at ieee.org     http://thebs413.blogspot.com
----------------------------------------------------------------------
The best things in life are NOT free - which is why life is easiest if
you save all the bills until you can share them with the perfect woman