How to continue a move if you hit ENTER instead of type CONTINUE?

[CRCinAU]

So I left the fsmove process run for ~25 hours, it did its thing, but it complained that I removed an old file from that drive while the move was in progress. It was an old .fstransform file that I didn't let go very long (hence didn't successfully copy a single file - I checked by mounting it).

Upon completion of the fsmove, it errored with:

2019-05-01 03:02:17 fsmove: progress: 98.4% done,  50.6 gigabytes still to move, estimated 25 minutes left
2019-05-01 03:08:02 fsmove: ERROR: failed to lstat() `/tmp/fstransform.mount.1097/.fstransform.loop.13621': No such file or directory
                   
2019-05-01 03:08:02 ERROR! fstransform: command '/usr/sbin/fsmove -- /tmp/fstransform.mount.1097 /tmp/fstransform.loop.1097 --exclude /tmp/fstransform.mount.1097/.fstransform.loop.1097' failed (exit status 254)
                          this is potentially a problem.
                          you can either quit now by pressing ENTER or CTRL+C,
                          
                          or, if you know what went wrong, you can fix it yourself,
                          then manually run the command '/usr/sbin/fsmove -- /tmp/fstransform.mount.1097 /tmp/fstransform.loop.1097 --exclude /tmp/fstransform.mount.1097/.fstransform.loop.1097'
                          (or something equivalent)
                          and finally resume this script by typing CONTINUE and pressing ENTER: 
2019-05-01 03:08:02 fstransform: exiting.

As I'd hit ENTER at some point in that terminal over the last 25 hours, it immediately quit.

I've mounted both the mount and loop mountpoints again, and I can confirm that all files except the excluded loop file have been moved to the loop filesystem.

So how to continue?

(and on another note, can we drop just a plain ENTER as being an ABORT?! especially if it may have been entered many, many hours prior and sat in a buffer since....

[CRCinAU]

I think I've managed to get things going again:

fsremap /dev/xvdb /tmp/fstransform.mount.1097/.fstransform.loop.1097 /tmp/fstransform.mount.1097/zerofile                                                         
fsremap: starting job 2, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.2'                                                                         
if this job is interrupted, for example by a power failure,
you CAN RESUME it with: fsremap -q --resume-job=2 -- /dev/xvdb
device length is 5.46 terabytes
analyzing device '/dev/xvdb', this may take some minutes ...
device effective block size = 4096
using reduced (32 bit) remapping algorithm
analysis completed: 3.01 terabytes must be relocated
primary-storage: located 2.24 terabytes (1728 fragments) usable in device (free, invariant, contiguous and aligned)                                                    
primary-storage: actually using 825.00 megabytes (1 fragment) from device
allocated 412.02 megabytes RAM as memory buffer
not creating secondary-storage, primary-storage is large enough
primary-storage is 825.00 megabytes, initialized and mmapped() to contiguous RAM
unmounting device '/dev/xvdb'... command: /bin/umount /dev/xvdb
successfully unmounted device '/dev/xvdb'
everything ready for in-place remapping, this is your LAST chance to quit.
WARN: press ENTER to proceed, or CTRL+C to quit

starting in-place remapping. this may take a LONG time ...
filling storage by moving 825.00 megabytes from device ...
storage filled
progress:  0.0% done,   3.0 terabytes still to relocate
moving 206.15 gigabytes from device to target ...

Disks are whirring away - so that's a good sign.

I would really feel better if the default of ENTER to ABORT would be changed to just loop and accept either Ctrl + C or CONTINUE....

[cosmos72]

You solved it by yourself :)

About changing ENTER to simply display the prompt again instead of aborting: it's definitely useful, thanks for the idea

[CRCinAU]

Yep - almost there:

progress: 87.3% done, 393.6 gigabytes still to relocate, estimated  3 hours left
moving 26.14 gigabytes from device to target ...

About changing ENTER to simply display the prompt again instead of aborting: it's definitely useful, thanks for the idea

Yeah - that's why I didn't close this issue.... It's almost standard to do the "Is that terminal still alive? enter enter enter" type thing - if you're not aware in advance that an enter will abort on errors, this leads to a "Oh crap, now what do I do?" moment. I had this myself :)

It took me ~45 minutes or so do hunt down the README.fsremap after I figured out what the fstransform script actually does and get it going again. Hopefully I've done it right... The figures it quotes seems to say so.

I've just been careful not to hit enter - as I don't know if that will auto-ack an abort if there's an error either ;)

[CRCinAU]

I'm a little concerned with the results of this scan - going from XFS -> ext4, the final fsck after fsremap finished is showing quite a few errors:

.....

Unattached inode 106642510                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642510 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642519                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642519 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642522                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642522 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642540                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642540 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642546                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642546 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642561                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642561 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642562                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642562 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642564                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642564 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642565                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642565 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642568                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642568 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642591                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642594 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642596                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642596 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642612                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642612 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642613                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642613 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642623                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642623 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached zero-length inode 106642626.  Clear? yes                                                                                                                                                                                           
                                                                                                                                                                                                                                              
Unattached inode 106642632                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642632 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642639                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642639 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642641                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642641 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642647                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642647 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        

Unattached inode 106642662
Connect to /lost+found? yes

Inode 106642662 ref count is 2, should be 1.  Fix? yes

Unattached inode 106642679                                                                                                                                                                                                          [164/1911]
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642679 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642684                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642684 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642686                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642686 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642695                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642695 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642701                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642701 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642709                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642709 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642714                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642714 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642715                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642715 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642718                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642718 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642729                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642729 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642733                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642733 ref count is 2, should be 1.  Fix? yes

Unattached inode 106642737                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642737 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642748                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642748 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642751                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642751 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642763                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642763 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642765                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642765 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642782                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642782 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642797                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642797 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642809                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642809 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642815                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642815 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642818                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642818 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642830                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642830 ref count is 2, should be 1.  Fix? yes

Unattached inode 106642833                                                                                                                                                                                                           [54/1911]
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642833 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642859                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642859 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642862                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642862 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642867                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642867 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642868                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642868 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642888                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642888 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached zero-length inode 106642895.  Clear? yes                                                                                                                                                                                           
                                                                                                                                                                                                                                              
Unattached inode 106642947                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642947 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106642992                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106642992 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106643006                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106643006 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106643019                                                                                                                                                                                                                    
Connect to /lost+found? yes                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Inode 106643019 ref count is 2, should be 1.  Fix? yes                                                                                                                                                                                        
                                                                                                                                                                                                                                              
Unattached inode 106643027                                                                                                                                                                                                                    
Connect to /lost+found? yes

Inode 106643027 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643037
Connect to /lost+found? yes

Inode 106643037 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643053
Connect to /lost+found? yes

Inode 106643053 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643054
Connect to /lost+found? yes

Inode 106643054 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643055
Connect to /lost+found? yes

Inode 106643055 ref count is 2, should be 1.  Fix? yes

Unattached zero-length inode 106643067.  Clear? yes

Unattached inode 106643070
Connect to /lost+found? yes

Inode 106643070 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643077
Connect to /lost+found? yes

Inode 106643077 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643080
Connect to /lost+found? yes

Inode 106643080 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643081
Connect to /lost+found? yes

Inode 106643081 ref count is 2, should be 1.  Fix? yes

Unattached inode 106643083
Connect to /lost+found? yes

Inode 106643083 ref count is 2, should be 1.  Fix? yes

Pass 5: Checking group summary information

/dev/xvdb: ***** FILE SYSTEM WAS MODIFIED *****
/dev/xvdb: 155480/183132160 files (5.7% non-contiguous), 819968052/1465037616 blocks

This left me with ~1800 files in /lost+found. I'm guessing this isn't a good thing - and the hard part now is trying to figure out what those files even are.

There were no errors in the fsck of the image file, nor the fsremap.

[cosmos72]

What version of fstransform/fsremap are you using?
Versions up to 0.9.3 have a bug that may cause filesystem corruption is you use a small RAM buffer (which seems to be your case, I saw a ~460MB RAM buffer somewhere in your logs) - see issue #13
The bug is fixed in 0.9.3.patch1 and in 0.9.4

[CRCinAU]

I'm using the version in Fedora 30 - packaged as: fstransform-0.9.3-4.fc30.x86_64

Looks like it hasn't changed in a while:

rpm -q --changelog fstransform | less
* Thu Jan 31 2019 Fedora Release Engineering <[email protected]> - 0.9.3-4
- Rebuilt for https://fedoraproject.org/wiki/Fedora_30_Mass_Rebuild

* Fri Jul 13 2018 Fedora Release Engineering <[email protected]> - 0.9.3-3
- Rebuilt for https://fedoraproject.org/wiki/Fedora_29_Mass_Rebuild

* Wed Feb 07 2018 Fedora Release Engineering <[email protected]> - 0.9.3-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_28_Mass_Rebuild

* Wed Aug 23 2017 Björn Esser <[email protected]> - 0.9.3-1
- Initial import (rhbz#1484166)

* Tue Aug 22 2017 Björn Esser <[email protected]> - 0.9.3-0.1
- Initial rpm release (rhbz#1484166)

I'll open up a bug report and get Fedora to update it.

EDIT: Bug report here: https://bugzilla.redhat.com/show_bug.cgi?id=1705564

[cosmos72]

Yes, thanks.
Actually it's safer to upgrade to 0.9.3.patch1, because 0.9.4 received significantly less testing (read: may have new bugs)

[CRCinAU]

Ok, I'll add this to the report (can't edit it).

[CRCinAU]

On the same topic of data corruption, is there a way to see what files got corrupted? I've found a few things that are dead - so far nothing important (yay) - but I'm not sure if I've found it all...

[cosmos72]

First of all, sorry for the corruption.
I tested fstransform quite heavily, but it's a complex program doing risky tricks with file systems - hence the big fat warnings everywhere.

The corruption bug #13 works like this: if you have a very large contiguous area on disk to copy (larger than the RAM buffer), only an initial fragment equal to the RAM buffer size will be copied - the rest will be missing, i.e. whatever happens to be already on disk in the destination area will remain there instead of being written over.

This happens at the block level, so large filesystem metadata blocks can get corrupted - thus the large number of fsck error. For files, it's not so common that two files are in consecutive blocks, but it may happen.
So I'd first run something like find -type f -print0 | xargs -0 file on the disk, to check for corrupted blocks at the beginning of each file (check for the output <FILENAME>: data i.e. contents that file does not recognize).
Corruption can also happen in the middle of a file, but that's harder to check.

[CRCinAU]

First of all, sorry for the corruption.

All good - its a bit of a pain - but the majority at the moment have been backups of stuff that it doesn't really matter if it is dead or not - ie I just blow away that entire backup.

My DVD collection is mostly mkv - so I'm using mkvmerge and setting its output to /dev/null to try and validate the bitstream throughout the file. That's pretty quick and seems to be a good detection for a corrupted file. I have nothing against trashing the corrupted ones - and if I can be bothered, re-ripping / encoding.

I haven't started validating the software ISOs - but they're mostly stuff like the Windows 10 ISO which can be downloaded from MS again anyway - so again, no great loss - just a bit annoying. There's Fedora ISOs, RHEL ISOs etc - same deal - nothing a redownload won't fix.

I did notice that a lot of photos backed up to that array seemed to survive - but the file was pretty much either empty, or with enough dead metadata that it couldn't decode. Checking with file *.jpg | grep 'data' seems to pick up most. These are all stored in Google Photos anyway - so again, nothing major lost.

I did end up moving away from that ext4 RAID6 to md RAID6 + btrfs though - which was my end goal... XFS -> ext4 -> BTRFS - as the btrfs-convert didn't want to work either :\

I ended up just rsync'ing everything to a borrowed 6Tb disk, rehashing the array, then rsync'ing back. So now just want to try and find as much of the corrupt stuff as I can.