Fedora SMP dual core, dual AMD 64 processor system

[Bryan J. Smith]

On Tue, 2005-08-16 at 17:39 -0700, Bill Broadley wrote:
> Why?

Please point out a _true_, intelligent RAID card for PCIe?

The only one I know of is the LSI Logic MegaRAID 320-3E.  And even it is
using the Intel IOP332, which is really a IOP331 with an internal bridge
from PCI-X to PCIe.

Until the Broadcom BCM8603 and other microntroller/ASIC solutions
appear, there are virtually _no_ intelligent RAID cards out there for
PCIe.

> Correct, although how many servers really need more than 1GB/sec,
> thats quite a bit even with say 16 drives hooked up.

That's if you _have_ PCI-X (or 0.5GBps out of 64-bit PCI 66MHz).
If you don't, all you get is 133MBps of shared, legacy 32-bit PCI.

I have seen too many small system assemblers and consultants putting in
PCI64/PCI-X storage controllers on the shared, legacy 32-bit PCI.

That wasn't even good 6 years ago!  ;->

> Intel makes reliable raid controllers and many a raid card uses them to
> provide what the market wants.

I think you mean Intel makes reliable RAID microcontrollers.  Yes, the
newer XScale (IOP32x/33x) are great.  I wouldn't say otherwise.

[ BTW, the older i960 (IOP30x) are rather lackluster today though.  But
people still buy products based on them, and wonder why they can't break
50-60MBps. ]

> Seems rather strange to advocate pci-x just because a popular pci-e
> solution uses a bridge.

No, what I said is that it's the _only_ solution on the market.  There
are _0_ others.  My comments on its being a hack were just added
comment, not my main argument.

> The migration is happening there are both AGP video cards with a pci-e
> bridge, and pci-e video cards with an agp bridge... who cares?

I'm not talking about video cards, I'm talking about _server_ I/O.

And, BTW, there _are_ performance/reliability issues with the AGP-PCIe
bridges -- just FYI.  On a desktop, that's not an issue.  On a server,
that can be.

> Even with video cards that are much more sensitive
> to bandwidth and latency there is no practical difference.

Again, I don't care about reliability of a desktop.  I'm talking a
server.  There is a _darth_ of PCIe storage controllers, and the only
one I know of isn't even native.

I'm _not_ saying the LSI Logic MegaRAID 320-2E is a "bad" solution.  I
said it's the _only_ solution available, and it's not even native.

> Keep in mind the IOP has an internal bridge from what I can tell, the
> video cards I'm familiar with have an external bridge.

Please stop talking about video cards, they have *0* to do with what I'm
talking about.

Again, there is virtually _no_ RAID solution for PCIe other than the LSI
Logic MegaRAID 320-2E right now.  That means either you have to pick it,
or you need to get a mainboard 

> Personally I'd rather have JBOD, I've yet to see a Raid controller
> faster than software RAID.

I'm not going there, but I disagree.  I don't like duplicating RAID-1
streams over the interconnect, and I definitely don't like cramming all
my disk I/O through my CPU interconnect to do RAID-5 (it's not the XOR
calculations that cost you performance, it's the tie-up of your
interconnect).

> I also like the standardized interface so if I have a few dozen
> servers I don't have to track the functionality I want via different
> command line tools, serial ports, web interfaces, front panels, and
> even custom window interfaces.

??? What solutions have you been using ???

I've been using 3Ware Escalade ASIC-based and StrongARM/X-Scale based
MegaRAID intelligent RAID solutions.  I _avoid_ external subsystems
(except for FC-AL), and I avoid vendors who don't support Linux.  3Ware
and Mylex/LSI have had excellent track records with Linux support, as
well as volume compatibility with newer cards.

> Not to mention the biggie, what happens if the raid card dies... being
> able to migrate to a random collection of hardware can be quite useful
> in an emergency.

Well, if you have standardized on 3Ware for your ATA, it's a no brainer,
you use another card.  Mylex/LSI have also been quite good on the SCSI
side.

I find this argument insignificant _if_ you choose a good vendor with a
solid history of volume upgrade path.  It's far better than dealing with
all the MD/LVM issues in Linux, no offense, but damn that's biting a big
chunk of my ass off too many times, and I just won't trust it.  ;->

> I bought one, many vendors sell them, what is the big deal?  Megaraid,
> tekram, lsi-logic, promise, even straight from intel if you want.
> Of course many more are coming from the likes of ICP-vortex, Adaptec,
> and just about anyone who wants to relabel and market an adapter in
> this space,

What PCIe RAID cards (and I'm not talking software driver RAID) don't I
know about?  The only one I know of is the LSI Logic MegaRAID 320-2E.

So far, all you've talked about is video cards.

> Why?  Slower, higher latency, lower scaling, and lower bandwidth.

Whoa!  Higher latency?  Not true at all.  Serial can increase latency
over parallel, but it makes up for it in dedicated throughput.  Don't
mix concepts, there are some trade-offs with serial (but they are
typically overcome with better designs).

> Not to mention the chipset quality has gone up.  I.e. even the same chip (pci-x
> or pci-e with or without a bridge) tends to be faster/lower latency
> under pci-e than pci-x.  From what I can tell it's mostly the quality
> of the new nvidia chipset that is the difference.  Interconnect vendors
> (again more sensitive to latency and bandwidth) are quite excited to be
> posting new numbers with the nvidia chipset.
> Not that it matters that much for a collection of 8-16 30-60MB disks
> on a server.

Again, what PCIe storage cards don't I know about?

-- 
Bryan J. Smith     b.j.smith at ieee.org     http://thebs413.blogspot.com
----------------------------------------------------------------------
The best things in life are NOT free - which is why life is easiest if
you save all the bills until you can share them with the perfect woman