Confirmed premature hard disk failure
Callum Lerwick
seg at haxxed.com
Wed Dec 31 01:21:26 UTC 2008
Now one occurrence doesn't prove it was the Load_Cycle_Count problem
that killed the drive, but this drive is definitely prematurely failed.
This is going to be spammy, I think it is important to document this so
we can all learn from it...
The drive I posted about a while back:
https://www.redhat.com/archives/fedora-devel-list/2008-July/msg00426.html
Is now definitely failing as of about 3-4 weeks ago. It's at the stage
where the drive still seems to be mostly readable, but IO bandwidth has
dropped like a rock. Booting Fedora takes like 20 minutes. Same thing
two previous failed drives did.
The drive has a manufacture date of May 2007. I have no reason to
believe the drive has been mistreated, no more than typical laptop wear
and tear. It is a "desktop replacement" laptop and is rarely
transported.
Fortunately /home seems to be readable. Media errors show up in the
syslog, but no IO errors are turning up in user space. The WinXP
partition seems to be trashed, however. So much for my wife's Half Life
2 and Spore savegames...
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
00 08 59 f1
sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
end_request: I/O error, dev sda, sector 547313
ata1: EH complete
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Full smartctl -a output, after salvaging some data:
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Scorpio family
Device Model: WDC WD600VE-11KWT0
Serial Number: WD-WXEY06009078
Firmware Version: 01.03K01
User Capacity: 60,011,642,880 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Tue Dec 30 17:09:36 2008 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (3180) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 44) minutes.
Conveyance self-test routine
recommended polling time: ( 6) minutes.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 198 198 051 Pre-fail Always - 5472
3 Spin_Up_Time 0x0003 150 119 021 Pre-fail Always - 1466
4 Start_Stop_Count 0x0032 097 097 000 Old_age Always - 3359
5 Reallocated_Sector_Ct 0x0033 197 197 140 Pre-fail Always - 19
7 Seek_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0
9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 6564
10 Spin_Retry_Count 0x0012 092 085 051 Old_age Always - 22
11 Calibration_Retry_Count 0x0012 100 100 051 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 359
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 350
193 Load_Cycle_Count 0x0032 061 061 000 Old_age Always - 419536
194 Temperature_Celsius 0x0022 087 071 000 Old_age Always - 56
196 Reallocated_Event_Count 0x0032 196 196 000 Old_age Always - 4
197 Current_Pending_Sector 0x0012 192 190 000 Old_age Always - 86
198 Offline_Uncorrectable 0x0010 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0009 200 200 051 Pre-fail Offline - 3
SMART Error Log Version: 1
ATA Error Count: 2803 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 2803 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 18 f1 41 e8 e2 Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 40 c9 41 e8 02 08 01:07:07.212 READ DMA
27 00 00 00 00 00 00 08 01:07:07.211 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 00 0a 01:07:07.208 IDENTIFY DEVICE
ef 03 45 00 00 00 00 0a 01:07:07.206 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 00 08 01:07:07.205 READ NATIVE MAX ADDRESS EXT
Error 2802 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 18 f1 41 e8 e2 Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 40 c9 41 e8 02 08 01:07:04.430 READ DMA
27 00 00 00 00 00 00 08 01:07:04.429 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 00 0a 01:07:04.427 IDENTIFY DEVICE
ef 03 45 00 00 00 00 0a 01:07:04.427 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 00 08 01:07:04.426 READ NATIVE MAX ADDRESS EXT
Error 2801 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 18 f1 41 e8 e2 Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 40 c9 41 e8 02 08 01:07:01.254 READ DMA
27 00 00 00 00 00 00 08 01:07:01.254 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 00 0a 01:07:01.251 IDENTIFY DEVICE
ef 03 45 00 00 00 00 0a 01:07:01.248 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 00 08 01:07:01.248 READ NATIVE MAX ADDRESS EXT
Error 2800 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 18 f1 41 e8 e2 Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 40 c9 41 e8 02 08 01:06:58.277 READ DMA
27 00 00 00 00 00 00 08 01:06:58.277 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 00 0a 01:06:58.274 IDENTIFY DEVICE
ef 03 45 00 00 00 00 0a 01:06:58.271 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 00 08 01:06:58.271 READ NATIVE MAX ADDRESS EXT
Error 2799 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 18 f1 41 e8 e2 Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 40 c9 41 e8 02 08 01:06:55.684 READ DMA
27 00 00 00 00 00 00 08 01:06:55.684 READ NATIVE MAX ADDRESS EXT
ec 00 00 00 00 00 00 0a 01:06:55.681 IDENTIFY DEVICE
ef 03 45 00 00 00 00 0a 01:06:55.678 SET FEATURES [Set transfer mode]
27 00 00 00 00 00 00 08 01:06:55.678 READ NATIVE MAX ADDRESS EXT
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: read failure 90% 6564 789370
# 2 Conveyance offline Completed: read failure 90% 6564 789370
# 3 Extended offline Completed: read failure 90% 6562 789370
# 4 Extended offline Completed: read failure 90% 6562 789369
# 5 Extended offline Completed: read failure 90% 6560 85755114
# 6 Extended offline Completed without error 00% 6455 -
# 7 Extended offline Interrupted (host reset) 60% 6455 -
# 8 Extended offline Completed without error 00% 4402 -
# 9 Extended offline Completed without error 00% 0 -
#10 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Self tests 1-4 were done just today. Test 5 was done when the drive
first started failing 3-4 weeks ago, after which the machine was kept
off until now. 6 and 7 are from when I first noticed
Current_Pending_Sector was nonzero, maybe a month or two ago. 8 is from
my post in July. 9 and 10 are from when the drive was new.
Things to note:
1) The SMART health status is PASSED. Just like every other failed drive
I've had...
2) None of the attributes are failed.
3) By the time the self tests start failing, it is too late, you are
already losing data.
We can not trust the drive to assess itself properly. If we ever get a
SMART desktop alert thing going, we MUST look directly at the
attributes, and come to our own conclusion.
Any questions before I RMA the drive?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-devel-list/attachments/20081230/a29a158d/attachment.sig>
More information about the fedora-devel-list
mailing list