From yusufg at outblaze.com Wed Nov 9 03:28:35 2005 From: yusufg at outblaze.com (Yusuf Goolamabbas) Date: Wed, 9 Nov 2005 11:28:35 +0800 Subject: dmesg shows stepping 0a for an AMD 248 which is not listed on the processor guide Message-ID: <20051109032835.GA29513@outblaze.com> Hi, On my AMD Opteron 248 box. I see the following CPU0: AMD Opteron(tm) Processor 248 stepping 0a AMD Opteron(tm) Processor 248 stepping 0a relevant stuff from /proc/cpuinfo processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 5 model name : AMD Opteron(tm) Processor 248 stepping : 10 I can't find this stepping number in the AMD Quick Processor Guide http://www.amd.com/qrgopteron1 I would like to find out if this stepping supports memory hole remapping in hardware or does it require software emulation. My understanding is that E-steppings of Opteron does memory hole remapping in hardware -- Yusuf Goolamabbas yusufg at outblaze.com From eugen at leitl.org Wed Nov 9 13:27:07 2005 From: eugen at leitl.org (Eugen Leitl) Date: Wed, 9 Nov 2005 14:27:07 +0100 Subject: Sun Fire X2100 Message-ID: <20051109132707.GU2249@leitl.org> I'm thinking about ordering a Sun Fire X2100. Has anyone here ran such a beast with Linux? Does the IPMI support work? Does the chipset support SATA NCQ under Linux, and have you used >250 GB hard drives with the machine? (I've got bitten with Sun hardware lock-in once). Is this hardware or software RAID? -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From avaalak at yahoo.com Sun Nov 13 19:31:13 2005 From: avaalak at yahoo.com (Doug Hussey) Date: Sun, 13 Nov 2005 11:31:13 -0800 (PST) Subject: BL25p V4 ES time issue Message-ID: <20051113193113.39756.qmail@web33114.mail.mud.yahoo.com> We are running HP Blades BL25p Dual AMD Opteron 252 with V4 2.6.9-11ELsmp kernel. (Stock Redhat Kernel). We are seeing an issue with the time clocks racing forward. Has anyone run into this bug? We have three blades and it is doing it on all of them. Thanks Doug From sbathe at gmail.com Mon Nov 14 03:49:06 2005 From: sbathe at gmail.com (Saurabh Bathe) Date: Mon, 14 Nov 2005 09:19:06 +0530 Subject: BL25p V4 ES time issue In-Reply-To: <20051113193113.39756.qmail@web33114.mail.mud.yahoo.com> References: <20051113193113.39756.qmail@web33114.mail.mud.yahoo.com> Message-ID: <43780932.6010100@gmail.com> Doug Hussey wrote: > We are running HP Blades BL25p Dual AMD Opteron 252 > with V4 2.6.9-11ELsmp kernel. (Stock Redhat Kernel). > > We are seeing an issue with the time clocks racing > forward. Has anyone run into this bug? We have three > blades and it is doing it on all of them. > Are you running NTP on the box? But yes there could also be an issue with kernel. Have a look at: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=165826 http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=312df5f1a1da780e084b328bcabb02a6dcd044c3 //Saurabh From avaalak at yahoo.com Mon Nov 14 06:19:32 2005 From: avaalak at yahoo.com (Doug Hussey) Date: Sun, 13 Nov 2005 22:19:32 -0800 (PST) Subject: BL25p V4 ES time issue In-Reply-To: <43780932.6010100@gmail.com> Message-ID: <20051114061932.2433.qmail@web33104.mail.mud.yahoo.com> --- Saurabh Bathe wrote: > Doug Hussey wrote: > > We are running HP Blades BL25p Dual AMD Opteron > 252 > > with V4 2.6.9-11ELsmp kernel. (Stock Redhat > Kernel). > > > > We are seeing an issue with the time clocks racing > > forward. Has anyone run into this bug? We have > three > > blades and it is doing it on all of them. > > > > Are you running NTP on the box? > But yes there could also be an issue with kernel. > > Have a look at: > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=165826 > http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=312df5f1a1da780e084b328bcabb02a6dcd044c3 > > //Saurabh > > -- > amd64-list mailing list > amd64-list at redhat.com > https://www.redhat.com/mailman/listinfo/amd64-list > Thanks for the links. I took a look and that appears to be our problem. We are using NTP. However, it looks like it might be a kernel issue. I am going to follow up with Redhat. Once again thanks for the links. cheers Doug From Craig.Miskell at agresearch.co.nz Mon Nov 14 18:57:37 2005 From: Craig.Miskell at agresearch.co.nz (Miskell, Craig) Date: Tue, 15 Nov 2005 07:57:37 +1300 Subject: BL25p V4 ES time issue Message-ID: > Message: 1 > Date: Sun, 13 Nov 2005 11:31:13 -0800 (PST) > From: Doug Hussey > Subject: BL25p V4 ES time issue > To: amd64-list at redhat.com > Message-ID: <20051113193113.39756.qmail at web33114.mail.mud.yahoo.com> > Content-Type: text/plain; charset=iso-8859-1 > > We are running HP Blades BL25p Dual AMD Opteron 252 > with V4 2.6.9-11ELsmp kernel. (Stock Redhat Kernel). > > We are seeing an issue with the time clocks racing > forward. Has anyone run into this bug? We have three > blades and it is doing it on all of them. Turn off cpuspeed e.g: # /etc/init.d/cpuspeed off It's a known issue with the CPU speed throttling on the Opterons - HP mentions it somewhere in release notes for the latest BIOS for the BL25ps. (or maybe I'm getting confused and cpuspeed is another issue with clock making them run slow... I forget ;-)) Craig Miskell ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. ======================================================================= From avaalak at yahoo.com Tue Nov 15 10:48:38 2005 From: avaalak at yahoo.com (Doug Hussey) Date: Tue, 15 Nov 2005 02:48:38 -0800 (PST) Subject: BL25p V4 ES time issue In-Reply-To: Message-ID: <20051115104838.16903.qmail@web33102.mail.mud.yahoo.com> "Miskell, Craig" wrote: > Message: 1 > Date: Sun, 13 Nov 2005 11:31:13 -0800 (PST) > From: Doug Hussey > Subject: BL25p V4 ES time issue > To: amd64-list at redhat.com > Message-ID: <20051113193113.39756.qmail at web33114.mail.mud.yahoo.com> > Content-Type: text/plain; charset=iso-8859-1 > > We are running HP Blades BL25p Dual AMD Opteron 252 > with V4 2.6.9-11ELsmp kernel. (Stock Redhat Kernel). > > We are seeing an issue with the time clocks racing > forward. Has anyone run into this bug? We have three > blades and it is doing it on all of them. Turn off cpuspeed e.g: # /etc/init.d/cpuspeed off It's a known issue with the CPU speed throttling on the Opterons - HP mentions it somewhere in release notes for the latest BIOS for the BL25ps. (or maybe I'm getting confused and cpuspeed is another issue with clock making them run slow... I forget ;-)) Craig Miskell ================================================================= That did the trick. There is more information on the HP website at EL050714_CW02 REVISION: 0 Thanks everyone for the help in this matter cheers Doug -- amd64-list mailing list amd64-list at redhat.com https://www.redhat.com/mailman/listinfo/amd64-list -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.j.smith at ieee.org Sat Nov 26 09:40:32 2005 From: b.j.smith at ieee.org (Bryan J. Smith) Date: Sat, 26 Nov 2005 04:40:32 -0500 Subject: Sun Fire X2100 -- nForce4 Ultra desktop chipset In-Reply-To: <20051109132707.GU2249@leitl.org> References: <20051109132707.GU2249@leitl.org> Message-ID: <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> On Wed, 2005-11-09 at 14:27 +0100, Eugen Leitl wrote: > I'm thinking about ordering a Sun Fire X2100. Has anyone > here ran such a beast with Linux? It's an nForce4 Ultra chipset, not worth the money IMHO. There are better 1U options IMHO. It only has PCIe. No PCI-X (if you want intelligent disk/redundancy). I'm sure there are plenty of vendors/integrators on this list that can hook you up with a nForce Professional or Broadcom ServerWorks chipset server solution in 1U that is much better, for about the same cost. > Does the IPMI support work? > Does the chipset support SATA NCQ under Linux, No. The nForce4 doesn't support NCQ under Windows either. It's basically an ATA controller, only capable of SATA. It does, however, offer hotplug and PM support. Driver is "nv_sata" (generic SCSI block driver, so devices will be /dev/sda). http://linux.yyz.us/sata/sata-status.html#nvidia You'll find the many controllers do _not_ do NCQ, and even the few that do have various issues, or just don't work under Linux with NCQ. BTW, NCQ is fairly overrated IMHO. For single disks, it depends on how well your OS flushes buffers. For multiple disks, you're now using your host to target multiple devices, and it would be far better for an intelligent host to control that (must like a real SCSI/SAS host does for its targets). The latest 3Ware 9550SX (PCI-X) and Areca 12xx (PCI-X or PCIe) series have on-board intelligence that handles multiple targeting of NCQ SATA devices. If you're really anal about NCQ, then you'll want to look to those cards. Both are supported in Linux. > and have you used >250 GB hard drives with the machine? Have a pair of WD 320GB drives on this very system (nForce4). There are no issues with supporting them at all. > (I've got bitten with Sun hardware lock-in once). These are standards-bsed AMD platforms. They even run Windows. Sun doesn't advertise that, but they do. > Is this hardware or software RAID? Software, of course! It's ATA FRAID (Fake RAID). 100% software driver, only 16-bit boot-time BIOS. Again, if you want intelligent hardware SATA raid with NCQ that works well under Linux, consider the 3Ware 9550SX (PCI-X) or Areca 12xx (PCI-X or PCIe). -- Bryan J. Smith b.j.smith at ieee.org http://thebs413.blogspot.com ------------------------------------------------------------------- For everything else *COUGH*commercials*COUGH* there's "ManningCard" From b.j.smith at ieee.org Sat Nov 26 09:42:49 2005 From: b.j.smith at ieee.org (Bryan J. Smith) Date: Sat, 26 Nov 2005 04:42:49 -0500 Subject: nForce4 motherboard recommendations In-Reply-To: <1130700916.3151.11.camel@gimly.scherco.local> References: <8767947e0510292056n4eaa0369scff204bd4043107b@mail.gmail.com> <1130700916.3151.11.camel@gimly.scherco.local> Message-ID: <1132998169.5023.257.camel@bert64.oviedo.smithconcepts.com> > ... As far as features, I would like SATA II (the 250 GB WD SATA II > drive is slightly cheaper than the SATA drive!) ... SATA-II is now a marketing term, much like USB 2.0. You have to have an SATA-II "IO" controller (aka SATA-IO) to get 3GHz, just like you have to have a EHCI USB2.0 controller to get 480Mbps. The SATA-II spec has become like USB 2.0, a set of features, not "required" features (such as speed). -- Bryan J. Smith b.j.smith at ieee.org http://thebs413.blogspot.com ------------------------------------------------------------------- For everything else *COUGH*commercials*COUGH* there's "ManningCard" From eugen at leitl.org Sat Nov 26 20:52:43 2005 From: eugen at leitl.org (Eugen Leitl) Date: Sat, 26 Nov 2005 21:52:43 +0100 Subject: Sun Fire X2100 -- nForce4 Ultra desktop chipset In-Reply-To: <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> Message-ID: <20051126205243.GI2249@leitl.org> On Sat, Nov 26, 2005 at 04:40:32AM -0500, Bryan J. Smith wrote: > On Wed, 2005-11-09 at 14:27 +0100, Eugen Leitl wrote: > > I'm thinking about ordering a Sun Fire X2100. Has anyone > > here ran such a beast with Linux? I've since found out that the beast runs Debian (and I presume, RedHat FC4/RHEL4) just fine. I'll try Solaris Express (and something based on OpenSolaris, eventually) as well, and run a few benchmarks. The zfs and the zones are a nice touch. > It's an nForce4 Ultra chipset, not worth the money IMHO. There are I've used nForce4 Ultra in my other (home) RAID server, precisely because it's a good performer, and inexpensive. So is the Sun Fire X2100 box, I'm not aware of another barebone with this quality for 600 EUR. > better 1U options IMHO. It only has PCIe. No PCI-X (if you want > intelligent disk/redundancy). The system has 2x GBit Ethernet, and in a pinch that PCIe can take an IB or Myrinet adaptor. The redundancy can be achieved with a distributed file system, or a AoE SATA enclosure. > I'm sure there are plenty of vendors/integrators on this list that can > hook you up with a nForce Professional or Broadcom ServerWorks chipset > server solution in 1U that is much better, for about the same cost. I would be very interested in a list of alternatives, in about the same price range (preferrably, with IPMI). I'm not aware of any. > BTW, NCQ is fairly overrated IMHO. For single disks, it depends on how I've since seen benchmarks which don't show more than 10% performance increase in best case, so NCQ is not that important. > well your OS flushes buffers. For multiple disks, you're now using your > host to target multiple devices, and it would be far better for an > intelligent host to control that (must like a real SCSI/SAS host does > for its targets). > > The latest 3Ware 9550SX (PCI-X) and Areca 12xx (PCI-X or PCIe) series > have on-board intelligence that handles multiple targeting of NCQ SATA > devices. If you're really anal about NCQ, then you'll want to look to > those cards. Both are supported in Linux. > > > and have you used >250 GB hard drives with the machine? > > Have a pair of WD 320GB drives on this very system (nForce4). There are > no issues with supporting them at all. I've gone with dual Hitachi Deskstar T7250s, which is a reasonable approximation to an enterprise SATA drive. > > (I've got bitten with Sun hardware lock-in once). > > These are standards-bsed AMD platforms. They even run Windows. Sun > doesn't advertise that, but they do. > > > Is this hardware or software RAID? > > Software, of course! It's ATA FRAID (Fake RAID). 100% software driver, > only 16-bit boot-time BIOS. > > Again, if you want intelligent hardware SATA raid with NCQ that works > well under Linux, consider the 3Ware 9550SX (PCI-X) or Areca 12xx (PCI-X > or PCIe). Linux MD RAID device works adequately. We'll see how zfs compares. -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From b.j.smith at ieee.org Sat Nov 26 23:49:36 2005 From: b.j.smith at ieee.org (Bryan J. Smith) Date: Sat, 26 Nov 2005 18:49:36 -0500 Subject: Sun Fire X2100 -- nForce4 Ultra desktop chipset In-Reply-To: <20051126205243.GI2249@leitl.org> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> <20051126205243.GI2249@leitl.org> Message-ID: <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> On Sat, 2005-11-26 at 21:52 +0100, Eugen Leitl wrote: > I've used nForce4 Ultra in my other (home) RAID server, precisely because > it's a good performer, and inexpensive. So is the Sun Fire X2100 box, > I'm not aware of another barebone with this quality for > 600 EUR. You'd be surprised. > The system has 2x GBit Ethernet, GbE on its own means nothing. > and in a pinch that PCIe can take an IB or Myrinet adaptor. Agreed. PCIe affords NIC and other options. > The redundancy can be achieved with a distributed file system, At a massive performance penalty. > or a AoE SATA enclosure. Don't tell me you've bought into their marketing too? ;-> CoRAID's performance is crap, as well as its features (or lackthereof). But let's talk beyond that. Redundancy is much cheaper internally. Or with a real SAN solution when you need failover. > I would be very interested in a list of alternatives, in about the > same price range (preferrably, with IPMI). I'm not aware of any. nForce Professional chipsets, as well as the Broadcom ServerWorks HT1000. Mainboards start at $200, both Socket-939 or 940. > I've since seen benchmarks which don't show more than 10% performance > increase in best case, so NCQ is not that important. Agreed. > I've gone with dual Hitachi Deskstar T7250s, which is a reasonable > approximation to an enterprise SATA drive. Actually, the Western Digital Raptor is an SATA version of the Hitachi UltraStar 10k. But I'd still consider 24x7 rated drives, even in the commodity capacities. > Linux MD RAID device works adequately. We'll see how zfs compares. Linux MD does the job adequately enough, I agree. On the SATA channels of the nForce4/Pro, at least they are on their own PCIe x1 channel. Not nearly as good as a PCI-X or PCIe hardware controller, but it does the job. -- Bryan J. Smith b.j.smith at ieee.org http://thebs413.blogspot.com ------------------------------------------------------------------- For everything else *COUGH*commercials*COUGH* there's "ManningCard" From eugen at leitl.org Sun Nov 27 09:02:27 2005 From: eugen at leitl.org (Eugen Leitl) Date: Sun, 27 Nov 2005 10:02:27 +0100 Subject: Sun Fire X2100 -- nForce4 Ultra desktop chipset In-Reply-To: <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> <20051126205243.GI2249@leitl.org> <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> Message-ID: <20051127090227.GM2249@leitl.org> On Sat, Nov 26, 2005 at 06:49:36PM -0500, Bryan J. Smith wrote: > You'd be surprised. Pointers? (Barebone with CPU and memory, no drives, good engineering, about 600 EUR). > > The system has 2x GBit Ethernet, > > GbE on its own means nothing. It means that I don't have a storage subsystem which can feed each machine at 200 MByte/s. My current HA NFS system (dual mini-ITX) maxes out at about two orders of magnitude less. > > The redundancy can be achieved with a distributed file system, > > At a massive performance penalty. Distributed file systems like Lustre and PVFS scale with the number of nodes. For my purposes (unified filestore in the rack, serving the world through a single 100 MBit Ethernet port) it's more than sufficient. I'm more concerned about power requirements than speed. Another must is redundancy, and remote management. > > or a AoE SATA enclosure. > > Don't tell me you've bought into their marketing too? ;-> > CoRAID's performance is crap, as well as its features (or lackthereof). I've never used it, but if it provides a couple of TBytes at 30-60 MByte/s throughput and failover for a reasonable price (say, 3 k$) that would be sufficient. > But let's talk beyond that. > Redundancy is much cheaper internally. Which is why I mentioned a distributed file system spanned over RAID 1 or RAID 5 volumes. > nForce Professional chipsets, as well as the Broadcom ServerWorks > HT1000. Mainboards start at $200, both Socket-939 or 940. I'm not interested in isolated barebones but complete systems, including CPU, some memory, sliding rails and IMPI or other out of band LOM. > > I've gone with dual Hitachi Deskstar T7250s, which is a reasonable > > approximation to an enterprise SATA drive. > > Actually, the Western Digital Raptor is an SATA version of the Hitachi > UltraStar 10k. But I'd still consider 24x7 rated drives, even in the > commodity capacities. I use Raptors in my desktop machines and home servers as the system drive but it is far too small for the storage volume/unit of rack space that I need. -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From b.j.smith at ieee.org Sun Nov 27 16:19:25 2005 From: b.j.smith at ieee.org (Bryan J. Smith) Date: Sun, 27 Nov 2005 11:19:25 -0500 Subject: Sun Fire X2100 -- nForce4 Ultra desktop chipset In-Reply-To: <20051127090227.GM2249@leitl.org> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> <20051126205243.GI2249@leitl.org> <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> <20051127090227.GM2249@leitl.org> Message-ID: <1133108365.5023.335.camel@bert64.oviedo.smithconcepts.com> On Sun, 2005-11-27 at 10:02 +0100, Eugen Leitl wrote: > Pointers? (Barebone with CPU and memory, no drives, good engineering, > about 600 EUR). Broadcom ServerWorks HT1000 chipset Socket-939 mainboards run only about $200, and have a PCI-X slot, and well as two (2) _server_ GbE NICs with 96KiB SRAM. Opteron 142-146 processors are only about $150-175. Non- registered ECC DDR SDRAM is really no premium over non-registered, non- ECC. It's basically the _same_ design as the Sun Fire x2100 -- Opteron and non-registered ECC -- only a heck of a _lot_better_ mainboard. As I mentioned, _several_ people here can help you with systems. > It means that I don't have a storage subsystem > which can feed each machine at 200 MByte/s. My current HA NFS > system (dual mini-ITX) maxes out at about two orders of magnitude > less. That's because it has a _joke_ of an I/O subsystem. The problem with Mini-ITX systems is the old, slow, shared PCI bus. You definitely will _not_ get past 100MBps on those. Only a few PCIe Mini-ITX mainboard are just coming out. That will at least help a bit. > Distributed file systems like Lustre and PVFS scale with > the number of nodes. For my purposes (unified filestore > in the rack, serving the world through a single 100 MBit > Ethernet port) it's more than sufficient. I'm more concerned > about power requirements than speed. Another must is redundancy, > and remote management. Sun's not offering any lower power than what you can get from others in an Opteron 1xxHE. > I've never used it, Exactly! Most people spewing CoRAID marketing aren't familiar with its difficiencies compared to a real SAN, or even multi-targettable SAS (if you just want shared storage in the same closet), which are faster (especially SAS). CoRAID was _smart_ to hit the trade show circuit. They've got everyone hyped up about AoE, which is not only 1 vendor, but rather "bare boned" in capabilities at this point. The drivers are still be perfected, with promised features still in development. Trying to use a CoRAID box as a failover is about the same as using a FireWire device. You don't have intelligence at the target to really do it, so it falls on GFS or something else, just like having _separate_ (not shared) storage. > but if it provides a couple of TBytes at 30-60 MByte/s throughput and > failover for a reasonable price (say, 3 k$) that would be sufficient. CoRAID doesn't provide true SAN-like failover. ;-> It requires you to run something like GFS, which is a massive amount of overhead. You might as well use local storage, it won't be any worse. And CoRAID is just slow, without much performance over iSCSI anyway (despite the marketing), with a lot less flexibility than iSCSI. > Which is why I mentioned a distributed file system > spanned over RAID 1 or RAID 5 volumes. > I'm not interested in isolated barebones but complete systems, > including CPU, some memory, sliding rails and IMPI or other > out of band LOM. ASL offers the Monarch 811x series starting at $700+: http://www.aslab.com/products/storage/monarch8115.html I would recommend the units with a 3Ware card for a little bit more: http://www.aslab.com/products/storage/monarch8114.html Again, I'm sure other system integrators on this list can help you further. > I use Raptors in my desktop machines and home servers as the system > drive but it is far too small for the storage volume/unit of rack > space that I need. I understand that, I just wanted to point out the enterprise capacities (18, 36, 73, etc...) are available in SATA. As I said, if you care about reliability in a commodity capacity (100, 200, 300, etc...), be sure to get a drive _explicitly_ tested and rated for 24x7 operation, not desktop 8x5 operation. Such models are the Seagate NL35, Western Digital Caviar RE and others. Not sure where the Deskstar T7K250 fall -- they might be Hitachi's 24x7 version of the Deskstar. -- Bryan J. Smith mailto:b.j.smith at ieee.org http://thebs413.blogspot.com ------------------------------------------ Some things (or athletes) money can't buy. For everything else there's "ManningCard." From eugen at leitl.org Sun Nov 27 18:55:02 2005 From: eugen at leitl.org (Eugen Leitl) Date: Sun, 27 Nov 2005 19:55:02 +0100 Subject: Sun Fire X2100 -- nForce4 Ultra desktop chipset In-Reply-To: <1133108365.5023.335.camel@bert64.oviedo.smithconcepts.com> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> <20051126205243.GI2249@leitl.org> <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> <20051127090227.GM2249@leitl.org> <1133108365.5023.335.camel@bert64.oviedo.smithconcepts.com> Message-ID: <20051127185502.GF2249@leitl.org> On Sun, Nov 27, 2005 at 11:19:25AM -0500, Bryan J. Smith wrote: > Broadcom ServerWorks HT1000 chipset Socket-939 mainboards run only about > $200, and have a PCI-X slot, and well as two (2) _server_ GbE NICs with > 96KiB SRAM. Opteron 142-146 processors are only about $150-175. Non- > registered ECC DDR SDRAM is really no premium over non-registered, non- > ECC. > > It's basically the _same_ design as the Sun Fire x2100 -- Opteron and > non-registered ECC -- only a heck of a _lot_better_ mainboard. > > As I mentioned, _several_ people here can help you with systems. Help as: offer a complete kit (sliding rails, IPMI, preassembled chassis only lacking memory and drives), warranty, and NBD support? My rack is more than two hours drive away from me (asuming I had a key for weekends and holidays -- I currently don't). I'd rather rely on redundancy and reliable components (this is no longer a Newisys box but native Sun engineering, which has a good reliability track). > That's because it has a _joke_ of an I/O subsystem. The problem with No, it's overhead of NFS, DRBD and limitations of Fast Ethernet. (Admittedly, 2-3 MByte/s according to bonnie++ is awful performance, even for DRBD+NFS). > Mini-ITX systems is the old, slow, shared PCI bus. You definitely will > _not_ get past 100MBps on those. Only a few PCIe Mini-ITX mainboard are > just coming out. That will at least help a bit. I've become somewhat disillusioned with rolling my own mini-ITX blades with Travla cases and rails. Mostly, it's lack of a decent airflow, especially for the faster/hotter drives, so packaging density suffers. Also, the only advantage is power efficiency, bought for by being somewhat expensive (cases and rails, my time) and having low absolute performance (still, enough for web, mail, HA NFS and a couple of other odd jobs). Here's a bonnie++ on the local filesystem (load zero, but on the network): Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP boron 1G 5470 81 11632 10 9652 18 7284 95 42007 33 186.1 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1210 68 +++++ +++ 1027 44 1153 67 +++++ +++ 778 41 boron,1G,5470,81,11632,10,9652,18,7284,95,42007,33,186.1,1,16,1210,68,+++++,+++,1027,44,1153,67,+++++,+++,778,41 Here's a single 36 GB Raptor (home server /, Ubuntu x86_64) in comparison: Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP bruell 2G 34425 63 52549 9 20808 4 37001 68 46619 4 224.2 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 3863 18 +++++ +++ 4150 23 3663 18 +++++ +++ 1532 7 bruell,2G,34425,63,52549,9,20808,4,37001,68,46619,4,224.2,0,16,3863,18,+++++,+++,4150,23,3663,18,+++++,+++,1532,7 Here's /home (RAID 5, ditto, ST3120213A and XFS): Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP bruell 2G 47025 85 61536 13 12784 2 24817 46 50324 6 302.0 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 473 3 +++++ +++ 457 2 375 2 +++++ +++ 302 2 bruell,2G,47025,85,61536,13,12784,2,24817,46,50324,6,302.0,0,16,473,3,+++++,+++,457,2,375,2,+++++,+++,302,2 > Sun's not offering any lower power than what you can get from others in > an Opteron 1xxHE. Yes, 230 W typical is more than I expected. We'll see how much the box will draw with the Hitachi drives, which are reasonably power-efficient. > > I've never used it, > > Trying to use a CoRAID box as a failover is about the same as using a > FireWire device. You don't have intelligence at the target to really do > it, so it falls on GFS or something else, just like having _separate_ > (not shared) storage. As I said, I intended to use a cluster FS, and a few machines in the rack. AoE is also quite economical if you roll your own (RAID 5 with one hot spare in a 4-drive SATA Linux box in 1U). > > but if it provides a couple of TBytes at 30-60 MByte/s throughput and > > failover for a reasonable price (say, 3 k$) that would be sufficient. > > CoRAID doesn't provide true SAN-like failover. ;-> It requires you to > run something like GFS, which is a massive amount of overhead. You Right now I'm more interested in Lustre or PVFS. It's still pretty academical at this point as I don't have enough hardware to prototype it. > might as well use local storage, it won't be any worse. And CoRAID is > just slow, without much performance over iSCSI anyway (despite the > marketing), with a lot less flexibility than iSCSI. > > > Which is why I mentioned a distributed file system > > spanned over RAID 1 or RAID 5 volumes. > > I'm not interested in isolated barebones but complete systems, > > including CPU, some memory, sliding rails and IMPI or other > > out of band LOM. > > ASL offers the Monarch 811x series starting at $700+: > http://www.aslab.com/products/storage/monarch8115.html Looks good. If I was in the U.S., I'd probably buy from them. > I would recommend the units with a 3Ware card for a little bit more: > http://www.aslab.com/products/storage/monarch8114.html > > Again, I'm sure other system integrators on this list can help you > further. > > > I use Raptors in my desktop machines and home servers as the system > > drive but it is far too small for the storage volume/unit of rack > > space that I need. > > I understand that, I just wanted to point out the enterprise capacities > (18, 36, 73, etc...) are available in SATA. > > As I said, if you care about reliability in a commodity capacity (100, > 200, 300, etc...), be sure to get a drive _explicitly_ tested and rated > for 24x7 operation, not desktop 8x5 operation. Such models are the I am very aware of this. I choose Hitachi (which are not explicitly rated enterprise class) because the Archive.org folks are using them, and reporting low failure rate (of course, my batch could be different). > Seagate NL35, Western Digital Caviar RE and others. Not sure where the > Deskstar T7K250 fall -- they might be Hitachi's 24x7 version of the > Deskstar. Hitachi officially doesn't have enterprise class products. Deathstars they're no longer. -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From b.j.smith at ieee.org Sun Nov 27 20:07:19 2005 From: b.j.smith at ieee.org (Bryan J. Smith) Date: Sun, 27 Nov 2005 15:07:19 -0500 Subject: Sun Fire X2100 -- nForce4 Ultra desktop chipset In-Reply-To: <20051127185502.GF2249@leitl.org> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> <20051126205243.GI2249@leitl.org> <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> <20051127090227.GM2249@leitl.org> <1133108365.5023.335.camel@bert64.oviedo.smithconcepts.com> <20051127185502.GF2249@leitl.org> Message-ID: <1133122039.5023.357.camel@bert64.oviedo.smithconcepts.com> On Sun, 2005-11-27 at 19:55 +0100, Eugen Leitl wrote: > Help as: offer a complete kit (sliding rails, IPMI, preassembled > chassis only lacking memory and drives), warranty, and NBD support? Yes. And ASL will as well. > No, it's overhead of NFS, DRBD and limitations of Fast Ethernet. > (Admittedly, 2-3 MByte/s according to bonnie++ is awful performance, > even for DRBD+NFS). Yes, it's absolute _crap_. Even with older P3s, or newer Pentium M (also a Mini-ITX option), you could get a lot better. I get 8-9MBps over fast Ethernet, over 40MBps sustained over GbE, 60+MBps with GbE using Jumbo frames. > I've become somewhat disillusioned with rolling my own mini-ITX blades > with Travla cases and rails. Mostly, it's lack of a decent airflow, especially > for the faster/hotter drives, so packaging density suffers. I know, I've had the Travla C156 in the marshes of Louisiana and Texas since October ... http://thebs413.blogspot.com/2005/09/travla-c156-12g512m40dvd-in-7x7x27.html They seem to be okay with the heat, but we're moving to a 400MHz, passively cooled ViA C3 in the near future. We only need to push through a sustained 5MBps _maximum_, so it's not an issue. When I need more power, like for real-time video streaming or portable NAS, I have been going i915 chipset with Pentium M. You can fit them with a pair of hard drives in 1U enclosure. I've been playing with the idea of (4) 2.5" SATA drives with an Areca RAID-10 (or even just software RAID-0+1) in one of these enclosures: http://www.commell.com.tw/Product/IPC/CMB-673.HTM I sure wish there were smaller, standards-based options for Opteron 1xx than a Socket-939 MicroATX mainboard (which is over 2x the size of Mini- ITX). Of course, I'd love to have some sort of embedded PCI-X while I'm at it. ;-> But hopefully there will be more PCIe options soon enough. > Also, the only advantage is power efficiency, bought for by being somewhat > expensive (cases and rails, my time) If you read my Blog, I found a reseller of MicroATX in California that can turnaround assemblies almost overnight or within 2 days. > and having low absolute performance (still, enough for web, mail, HA > NFS and a couple of other odd jobs). Web, maybe. But for NFS, not what I'm used to. I'm used to pushing over 50MBps over NFS. > Here's a bonnie++ on the local filesystem (load zero, but on the network): Load on the I/O is difficult to measure in Linux. At most, all you can measure is how often the CPU is bothered by I/O, not the actual efficiency/saturation of the I/O interconnect. Still, these numbers are horrendous. > As I said, I intended to use a cluster FS, and a few machines in the > rack. AoE is also quite economical if you roll your own (RAID 5 with > one hot spare in a 4-drive SATA Linux box in 1U). AoE is marketing. ;-> > Right now I'm more interested in Lustre or PVFS. It's still pretty > academical at this point as I don't have enough hardware to prototype it. For web and other Internet services. But for network file services? > Looks good. If I was in the U.S., I'd probably buy from them. As I said, there are probably some integrators on this list that are more local to you. > I am very aware of this. I choose Hitachi (which are not explicitly > rated enterprise class) because the Archive.org folks are using them, > and reporting low failure rate (of course, my batch could be different). Hitachi has some 24x7 rated units. They are preferred in many systems, including having an investment in Copan Systems: http://www.copansys.com/products/specifications.htm I highlighted them in an article on how storage has morphed into VTL: http://www.samag.com/documents/sam0509a/ > Hitachi officially doesn't have enterprise class products. Deathstars > they're no longer. I don't know about that, their 500GB Deskstar uses *5* commodity platters in a 1" form-factor. The first to do so in a _long_ time. I.e., not since the infamous 75GXP. But you're right, the 250GB you're using is a 80GB/platter drive with 2-3 platters. -- Bryan J. Smith mailto:b.j.smith at ieee.org http://thebs413.blogspot.com ------------------------------------------ Some things (or athletes) money can't buy. For everything else there's "ManningCard." From b.j.smith at ieee.org Tue Nov 29 10:58:53 2005 From: b.j.smith at ieee.org (Bryan J. Smith) Date: Tue, 29 Nov 2005 05:58:53 -0500 Subject: SuperMicro H8SSL-i (ServerWorks HT1000) -- WAS: Sun Fire X2100/nForce4 Ultra desktop In-Reply-To: <1133108365.5023.335.camel@bert64.oviedo.smithconcepts.com> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> <20051126205243.GI2249@leitl.org> <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> <20051127090227.GM2249@leitl.org> <1133108365.5023.335.camel@bert64.oviedo.smithconcepts.com> Message-ID: <1133261933.5023.543.camel@bert64.oviedo.smithconcepts.com> On Sun, 2005-11-27 at 11:19 -0500, Bryan J. Smith wrote: > Broadcom ServerWorks HT1000 chipset Socket-939 mainboards run only about > $200, and have a PCI-X slot, and well as two (2) _server_ GbE NICs with > 96KiB SRAM. Opteron 142-146 processors are only about $150-175. Non- > registered ECC DDR SDRAM is really no premium over non-registered, non- > ECC. FYI, it was an integrator on this list that turned me on to the SuperMicro H8SSL-i after I mentioned the Broadcom ServerWorks HT1000 chipset. Homepage for the SuperMicro H8SSL-i: http://www.supermicro.com/Aplus/motherboard/Opteron/HT1000/H8SSL-i.cfm Manual for the SuperMicro H8SSL-i: http://www.supermicro.com/manuals/motherboard/HT1000/MNL-H8SSL-i.pdf One thing that confuses me is the block diagram on page 1-8 (PDF page 14). It says the dual-GbE LAN (provided by the BCM5704*1* according to the same manual, page 1-8, PDF page 13) is on the PCI-X bus along with the slot, but that it runs at 133MHz. I was not aware that PCI-X [1.0] was capable of running more than 1 device at 133MHz. It could be that it slows down to 100MHz if you put in a PCI-X card. Of course, if you put in a legacy 64-bit at 66MHz 3.3V PCI card like a $125 3Ware Escalade 8006-2 (2-channel, true ASIC hardware SATA RAID) or $250 3Ware Escalade 8506-4 (4-channel), then it will run at 66MHz (should auto-detect, although jukmper JPXA1 lets you manually set both options of PCI-X 66MHz or legacy PCI 66MHz). But that is still 0.5GBps for NIC and storage -- 4x standard PCI or a PCIe x1 channel (although the PCIe x1 is bi-directional). According to the BCM5704*1* product brief, each NIC has its own RISC off-loading processor 16KiB of SRAM as well as dual 64KiB RAM packet buffer. You might be eyeing the $50-60 SysKonnect SK-9E21D*2* PCIe x1 NIC, it has a 48KiB RAM packet buffer (no RISC+SRAM off-load engine -- just a "large send off-load"). Even the $125-150+, server-designed PCIe x1 SK-9E21 or PCIe x4 SK-9E21 (PCI-X equivalent SK-9S21 and SK-9S22, respectively) doesn't seem to offer any off-load other than "large send off-load" (can't tell how good, the manual is in French or German and doesn't make any mention of SRAM), only 2x the RAM (per port), 96KiB. *1* Broadcom BCM5704 Product Brief (first page): http://www.broadcom.com/collateral/pb/5704C-PB04-R.pdf *2* SysKonnect SK-9E21D Manual (page 41, PDF page 39): http://www.skd.de/e_en/products/adapters/pcie_desktop/sk-9exxd/docu/Sk-9E21D_E.pdf Some resellers are claiming they are selling the SuperMicro H8SSL-i, although I thought SuperMicro was against that, and only provided it for resellers (largely because of their relationship with Intel). Both are out-of-stock: PC Super Deals: $244.65 http://www.pcsuperdeals.com/Productview.asp?ProductID=fd7ac3dd-0ee2-4ca1-9748-1c6c1b67fad4 Buy.COM: http://www.buy.com/retail/product.asp?sku=202106965 -- Bryan J. Smith mailto:b.j.smith at ieee.org http://thebs413.blogspot.com ------------------------------------------ Some things (or athletes) money can't buy. For everything else there's "ManningCard." From b.j.smith at ieee.org Tue Nov 29 11:30:06 2005 From: b.j.smith at ieee.org (Bryan J. Smith) Date: Tue, 29 Nov 2005 06:30:06 -0500 Subject: SuperMicro H8SSL-i (ServerWorks HT1000) -- WAS: Sun Fire X2100/nForce4 Ultra desktop In-Reply-To: <1133261933.5023.543.camel@bert64.oviedo.smithconcepts.com> References: <20051109132707.GU2249@leitl.org> <1132998033.5023.254.camel@bert64.oviedo.smithconcepts.com> <20051126205243.GI2249@leitl.org> <1133048976.5023.302.camel@bert64.oviedo.smithconcepts.com> <20051127090227.GM2249@leitl.org> <1133108365.5023.335.camel@bert64.oviedo.smithconcepts.com> <1133261933.5023.543.camel@bert64.oviedo.smithconcepts.com> Message-ID: <1133263807.5023.562.camel@bert64.oviedo.smithconcepts.com> On Tue, 2005-11-29 at 05:58 -0500, Bryan J. Smith wrote: > According to the BCM5704*1* product brief, each NIC has its own RISC > off-loading processor 16KiB of SRAM as well as dual 64KiB RAM packet > buffer. You might be eyeing the $50-60 SysKonnect SK-9E21D*2* PCIe x1 > NIC, it has a 48KiB RAM packet buffer (no RISC+SRAM off-load engine -- > just a "large send off-load"). Even the $125-150+, server-designed PCIe > x1 SK-9E21 or PCIe x4 SK-9E21 (PCI-X equivalent SK-9S21 and SK-9S22, > respectively) doesn't seem to offer any off-load other than "large send > off-load" (can't tell how good, the manual is in French or German and > doesn't make any mention of SRAM), only 2x the RAM (per port), 96KiB. > *1* Broadcom BCM5704 Product Brief (first page): > http://www.broadcom.com/collateral/pb/5704C-PB04-R.pdf > *2* SysKonnect SK-9E21D Manual (page 41, PDF page 39): > http://www.skd.de/e_en/products/adapters/pcie_desktop/sk-9exxd/docu/Sk-9E21D_E.pdf Actually, I was doing some research, and it seems that only LSO is supported in the BCM5704 firmware under Linux. And it seems that most "quality" TX/RX designs these days have 16KiB SRAM so they can handle up to 16KiB Jumbo frames. You only need to be careful about the NICs that have only 2-4KiB SRAM TX/RX that can't handle jumbo frames. -- Bryan J. Smith mailto:b.j.smith at ieee.org http://thebs413.blogspot.com ------------------------------------------ Some things (or athletes) money can't buy. For everything else there's "ManningCard."