Recover data from broken RAID 1 drive

peterPetersonling asked:

I need to save files from a Buffalo NAS Linkstation with two physical drives that is not working anymore (2TB each, Raid 1). Error message says:

The bad sectors in the hard drive may have reached a dangerous level.
Replace the hard drive.

I removed both drives from the NAS and tried to connect the drives externally to my computer via USB adapter using the following steps:

  1. mdadm --assemble --readonly /dev/md0 /dev/sdc6
  2. mount /dev/md0

For drive 1 this works fine. I have access to the files. BUT: the relevant data is not available on disk 1. It seems the files were not synchronized anymore.

Unfortunately these steps do not work for the second drive. I get the following error:

mdadm: no recogniseable superblock on /dev/sdc6 mdadm: /dev/sdc6 has
no superblock – assembly aborted

I am new to Linux and don’t really know what to do next. Is there a chance of getting the files from the crashed drive?


Some additional information:

lsblk:

sdc                       8:32   0   1,8T  0 disk  
├─sdc1                    8:33   0   977M  0 part  
├─sdc2                    8:34   0   4,8G  0 part  
├─sdc3                    8:35   0     1M  0 part  
├─sdc4                    8:36   0     1M  0 part  
├─sdc5                    8:37   0   977M  0 part  
└─sdc6                    8:38   0   1,8T  0 part  

dmesg:

[  867.209205]  sdc: sdc1 sdc2 sdc3 sdc4 sdc5 sdc6
[  867.210767] sd 3:0:0:0: [sdc] Attached SCSI disk
[  868.000588] sd 3:0:0:0: [sdc] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.000591] sd 3:0:0:0: [sdc] tag#17 Sense Key : Hardware Error [current] 
[  868.000594] sd 3:0:0:0: [sdc] tag#17 Add. Sense: Internal target failure
[  868.000597] sd 3:0:0:0: [sdc] tag#17 CDB: Read(10) 28 00 00 1e 90 08 00 00 08 00
[  868.000599] print_req_error: critical target error, dev sdc, sector 2002952
[  868.000634] sd 3:0:0:0: [sdc] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.000637] sd 3:0:0:0: [sdc] tag#18 Sense Key : Hardware Error [current] 
[  868.000641] sd 3:0:0:0: [sdc] tag#18 Add. Sense: Internal target failure
[  868.000644] sd 3:0:0:0: [sdc] tag#18 CDB: Read(10) 28 00 00 b7 28 80 00 00 08 00
[  868.000646] print_req_error: critical target error, dev sdc, sector 12003456
[  868.000700] sd 3:0:0:0: [sdc] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.000701] sd 3:0:0:0: [sdc] tag#16 Sense Key : Hardware Error [current] 
[  868.000702] sd 3:0:0:0: [sdc] tag#16 Add. Sense: Internal target failure
[  868.000703] sd 3:0:0:0: [sdc] tag#16 CDB: Read(10) 28 00 00 1e 8f f8 00 00 08 00
[  868.000704] print_req_error: critical target error, dev sdc, sector 2002936
[  868.000712] sd 3:0:0:0: [sdc] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.000713] sd 3:0:0:0: [sdc] tag#19 Sense Key : Hardware Error [current] 
[  868.000715] sd 3:0:0:0: [sdc] tag#19 Add. Sense: Internal target failure
[  868.000717] sd 3:0:0:0: [sdc] tag#19 CDB: Read(10) 28 00 e7 e6 5a 58 00 00 08 00
[  868.000718] print_req_error: critical target error, dev sdc, sector 3890633304
[  868.000731] sd 3:0:0:0: [sdc] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.000733] sd 3:0:0:0: [sdc] tag#20 Sense Key : Hardware Error [current] 
[  868.000734] sd 3:0:0:0: [sdc] tag#20 Add. Sense: Internal target failure
[  868.000736] sd 3:0:0:0: [sdc] tag#20 CDB: Read(10) 28 00 00 b7 30 00 00 00 08 00
[  868.000737] print_req_error: critical target error, dev sdc, sector 12005376
[  868.000747] sd 3:0:0:0: [sdc] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.000749] sd 3:0:0:0: [sdc] tag#25 Sense Key : Hardware Error [current] 
[  868.000751] sd 3:0:0:0: [sdc] tag#25 Add. Sense: Internal target failure
[  868.000753] sd 3:0:0:0: [sdc] tag#25 CDB: Read(10) 28 00 00 d5 bf f0 00 00 08 00
[  868.000753] print_req_error: critical target error, dev sdc, sector 14008304
[  868.658878] sd 3:0:0:0: [sdc] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.658884] sd 3:0:0:0: [sdc] tag#17 Sense Key : Illegal Request [current] 
[  868.658889] sd 3:0:0:0: [sdc] tag#17 Add. Sense: Invalid field in cdb
[  868.658894] sd 3:0:0:0: [sdc] tag#17 CDB: Read(10) 28 00 00 1e 90 08 00 00 08 00
[  868.658898] print_req_error: critical target error, dev sdc, sector 2002952
[  868.658911] Buffer I/O error on dev sdc2, logical block 1, async page read
[  868.659015] sd 3:0:0:0: [sdc] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.659021] sd 3:0:0:0: [sdc] tag#16 Sense Key : Illegal Request [current] 
[  868.659026] sd 3:0:0:0: [sdc] tag#16 Add. Sense: Invalid field in cdb
[  868.659032] sd 3:0:0:0: [sdc] tag#16 CDB: Read(10) 28 00 00 1e 8f f8 00 00 08 00
[  868.659036] print_req_error: critical target error, dev sdc, sector 2002936
[  868.659043] Buffer I/O error on dev sdc1, logical block 250111, async page read
[  868.659083] sd 3:0:0:0: [sdc] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.659088] sd 3:0:0:0: [sdc] tag#20 Sense Key : Illegal Request [current] 
[  868.659094] sd 3:0:0:0: [sdc] tag#20 Add. Sense: Invalid field in cdb
[  868.659099] sd 3:0:0:0: [sdc] tag#20 CDB: Read(10) 28 00 e7 e6 5a 58 00 00 08 00
[  868.659103] print_req_error: critical target error, dev sdc, sector 3890633304
[  868.659109] Buffer I/O error on dev sdc6, logical block 484578123, async page read
[  868.659131] sd 3:0:0:0: [sdc] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.659136] sd 3:0:0:0: [sdc] tag#21 Sense Key : Illegal Request [current] 
[  868.659141] sd 3:0:0:0: [sdc] tag#21 Add. Sense: Invalid field in cdb
[  868.659147] sd 3:0:0:0: [sdc] tag#21 CDB: Read(10) 28 00 00 b7 30 00 00 00 08 00
[  868.659151] print_req_error: critical target error, dev sdc, sector 12005376
[  868.659157] Buffer I/O error on dev sdc4, logical block 0, async page read
[  868.659177] Buffer I/O error on dev sdc5, logical block 250110, async page read
[  868.659195] Buffer I/O error on dev sdc3, logical block 16, async page read

My answer:


You’re in a bad spot here, because you think you had one drive fail. In reality, BOTH drives failed.

What happened here is that some time ago, the NAS noticed some sort of problem with one drive and kicked it out of the array. It was probably something quite minor, as the drive sounds like it is working. It should have generated an event at that time, but if you didn’t have the NAS configured to notify you by some method (e.g. email) then you might not have noticed, and thought everything was fine since it was still running (albeit degraded) on the second disk. If you had noticed at that time and fixed the problem, you would have been able to recover files today.

Today, your second disk experienced a catastrophic failure. It’s unlikely you’ll be able to recover anything from it, but you can try using dd_rescue to make an image of the disk which skips any bad sectors, then use the image to try to mount the array. If that fails, your last resort is a professional data recovery company. But before that, you should go to your backups.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.