[Tig] Can someone help me with my drive array?

[Bob Friesenhahn]

On Mon, 18 Feb 2008, Michael Russell wrote:
>  To maximize performance:
>    1) You want to have write-caching enabled on the drives.  Naturally, this
>       has the potential for data loss depending on the battery backup of
>       the RAID controller, drives and probably some other factors.
>       However, based on your results (ballpark 180MB/s write & 330 MB/s
>       read), I'd say this is one of your issues.  You should discuss this
>       issue with Sun to see what the potential for data loss is and whether
>       it's OK for your situation.

Yes.  There were several issues.  The StorageTek 2540 is an 
"enterprise" type product so that its primary requirement is to never 
lose any data.  It does non-volatile write caching, but since it has 
redundant controllers, it also caches writes in the other controller. 
This redundant caching was costing some performance.  Sun also 
provided me with a special tweak so that the drive array would report 
that the data was synced when the host sent a sync request. 
Otherwise it was actually flushing its non-volatile write cache.

I am using striped mirroring (similar to RAID 1+0) since I am too poor 
to afford the backup requirements (or risks) associated with RAID 0. I 
have the I/O paths balanced so that mirror pairs are on different 
controllers and FC paths.  This assures the most uniform load, and 
also helps assure that I don't lose data.

I am now up to 274MB/second for writes and 484MB/second for reads on a 
64GB file.  Performance seems much faster for smaller files (e.g. 
372MB/second for writes, and 1,712MB/second for reads on an 8GB file) 
since my system does very aggressive caching.

The performance I am getting now is considerably more than Sun's 
official benchmark data for this product.  In fact, the official 
benchmark (from April, 2007) only achieved 105MB/second for large file 
writes.

>    3) You want to make sure that the channel is not saturated when you
>       have other options.  For example, I'd guess that each of the drives
>       can do about 80 MB/s, so 12 drives could theoretically (very, very
>       theoretically) do about 960 MB/s in a RAID 0 configuration.  This, is

I think that disk IOPS are the limiting factor here rather than drive 
bandwidth. I am testing single-file read/writes so the I/O is very 
intense.  With more of a "multi-user" usage model, the overall 
throughput of this drive array goes way up.

>       of course more than a single 4Gb FC channel can handle.  However, if
>       you split the drives up so that 6 are on each of your 2 channels,
>       you'll avoid this problem.  Depending on the RAID controller /
>       switch / box etc., this may be anywhere from simple to
>       impossible to do.

That is exactly what I did!

>  I hope some of this helps.

It seems like you know a whole lot more about this topic than you 
initially let on. :-)

Bob
======================================
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/