Conversion failed with fsmove error

[aciavolino]

17:18:46 ERROR! fstransform: command '/sbin/fsmove -- /tmp/fstransform.mount.12641 /tmp/fstransform.loop.12641 --exclude /tmp/fstransform.mount.12641/.fstransform.loop.12641' failed (exit status 240)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/sbin/fsmove -- /tmp/fstransform.mount.12641 /tmp/fstransform.loop.12641 --exclude /tmp/fstransform.mount.12641/.fstransform.loop.12641'
            (or something equivalent)

[cosmos72]

Hi @aciavolino
can you post the lines before this error?

It's unusual for fsmove to fail, but it can happen, and it should have logged the reason.
Also, knowing what conversion you were doing and the partition size and usage % could help.

[aciavolino]

[root@ctstageextweb2vm ~]# fstransform /dev/sda1 ext4
fstransform: starting version 0.9.3, checking environment
fstransform: checking for which... '/bin/which'
fstransform: checking for expr... '/bin/expr'
fstransform: checking for id... '/bin/id'
fstransform: parsing command line arguments
fstransform: checking for stat... '/bin/stat'
fstransform: checking for mkfifo... '/bin/mkfifo'
fstransform: checking for blockdev... '/sbin/blockdev'
fstransform: checking for losetup... '/sbin/losetup'
fstransform: checking for fsck... '/sbin/fsck'
fstransform: checking for mkfs... '/sbin/mkfs'
fstransform: checking for mount... '/bin/mount'
fstransform: checking for umount... '/bin/umount'
fstransform: checking for mkdir... '/bin/mkdir'
fstransform: checking for rmdir... '/bin/rmdir'
fstransform: checking for rm... '/bin/rm'
fstransform: checking for dd... '/bin/dd'
fstransform: checking for sync... '/bin/sync'
fstransform: checking for fsmove... '/sbin/fsmove'
fstransform: checking for fsremap... '/sbin/fsremap'
fstransform: checking for fsck(source file-system)... '/sbin/fsck'
fstransform: checking for fsck(target file-system)... '/sbin/fsck'
fstransform: looking for optional commands
fstransform: checking for sleep... '/bin/sleep'
fstransform: checking for date... '/bin/date'
08:07:01 fstransform: environment check passed.
08:07:01 fstransform: saving output of this execution into /var/tmp/fstransform/ fstransform.log.22858
08:07:01 fstransform: preparing to transform device '/dev/sda1' to file-system t ype 'ext4'
08:07:01 fstransform: device '/dev/sda1' not found in the output of command /bin /mount, assuming it is not mounted
08:07:01 fstransform: device is now mounted at '/tmp/fstransform.mount.22858' wi th file-system type 'xfs'
08:07:01 fstransform: device raw size = 1073741824 bytes

08:07:01 fstransform: WARNING: possibly stale fstransform loop files found insid e device '/dev/sda1',
maybe they can be removed? list of files found:

                  /tmp/fstransform.mount.22858/.fstransform.loop.11631 /tmp/                                                                                                                                                             fstransform.mount.22858/.fstransform.loop.11837 /tmp/fstransform.mount.22858/.fs                                                                                                                                                             transform.loop.12027 /tmp/fstransform.mount.22858/.fstransform.loop.12252 /tmp/f                                                                                                                                                             stransform.mount.22858/.fstransform.loop.12641

08:07:01 fstransform: creating sparse loop file '/tmp/fstransform.mount.22858/.f stransform.loop.22858' inside device '/dev/sda1'...
08:07:01 dd: 1+0 records in
08:07:01 dd: 1+0 records out
08:07:01 dd: 1 byte (1 B) copied, 0.0003222 s, 3.1 kB/s
08:07:01 fstransform: device file-system block size = 4096 bytes
08:07:01 fstransform: device usable size = 1073741824 bytes
08:07:01 dd: 1+0 records in
08:07:01 dd: 1+0 records out
08:07:01 dd: 1 byte (1 B) copied, 0.0003261 s, 3.1 kB/s
08:07:01 fstransform: connected loop device '/dev/loop0' to file '/tmp/fstransfo rm.mount.22858/.fstransform.loop.22858'
08:07:01 fstransform: formatting loop device '/dev/loop0' with file-system type 'ext4'...
08:07:02 fstransform: mounting loop device '/dev/loop0' on '/tmp/fstransform.loo p.22858' ...
08:07:02 fstransform: loop device '/dev/loop0' mounted successfully.
08:07:02 fstransform: preliminary steps completed, now comes the delicate part:
08:07:02 fstransform: fstransform will move '/dev/sda1' contents into the loop f ile.

08:07:02 fstransform: WARNING: THIS IS IMPORTANT! if either the original device '/dev/sda1'
or the loop device '/dev/loop0' become FULL,

                   YOU  WILL  LOSE  YOUR  DATA !

                  fstransform checks for enough available space,
                  in any case it is recommended to open another terminal, ty                                                                                                                                                             pe
                    watch df /dev/sda1 /dev/loop0
                  and check that both the original device '/dev/sda1'
                  and the loop device '/dev/loop0' are NOT becoming full.
                  if one of them is becoming full (or both),
                  you MUST stop fstransform with CTRL+C or equivalent.

                  this is your chance to quit.
                  press ENTER to continue, or CTRL+C to quit:

08:07:10 fstransform: moving '/dev/sda1' contents into the loop file.
08:07:10 fstransform: this may take a long time, please be patient...
08:07:10 fsmove: ERROR: failed to remove source directory `/tmp/fstransform.moun t.22858/efi': Device or resource busy

08:07:10 ERROR! fstransform: command '/sbin/fsmove -- /tmp/fstransform.mount.228 58 /tmp/fstransform.loop.22858 --exclude /tmp/fstransform.mount.22858/.fstransfo rm.loop.22858' failed (exit status 240)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/sbin/fsmove -- /tmp/fstransform.                                                                                                                                                             mount.22858 /tmp/fstransform.loop.22858 --exclude /tmp/fstransform.mount.22858/.                                                                                                                                                             fstransform.loop.22858'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing E                                                                                                                                                         NTER:

[aciavolino]

I am trying to convert the boot partition from xfs to ext4

[cosmos72]

The error is

08:07:10 fsmove: ERROR: failed to remove source directory `/tmp/fstransform.mount.22858/efi': Device or resource busy

Fstransform does not support converting file systems which contain mount points for other partitions: the usual solution is to unmount them.
Actually, this is a strange error because /dev/sda1 was mounted by fstranform itself, so it should not contain mount points for other partitions. What does ls -l efi show when run inside the filesystem you are converting? (you will need to mount it somewhere)

Also, the following warning means several previous conversion attempts failed:

08:07:01 fstransform: WARNING: possibly stale fstransform loop files found inside device '/dev/sda1',
maybe they can be removed? list of files found:
    /tmp/fstransform.mount.22858/.fstransform.loop.11631 /tmp/fstransform.mount.22858/.fstransform.loop.11837 /tmp/fstransform.mount.22858/.fstransform.loop.12027 /tmp/fstransform.mount.22858/.fstransform.loop.12252 /tmp/fstransform.mount.22858/.fstransform.loop.12641

You really need to undo such previous conversion attempts, by loop-mounting those files and copying back their contents to the partition you are converting. Or, if they are empty, by deleting such files.

[aciavolino]

Ok so I did unmount that drive and it still gives the error. I will look into removing those. How exactly do I loop mount them?

[cosmos72]

Supposing /dev/sda1 is mounted somewhere - for example in /tmp/fstransform.mount.22858 - you can loop-mount the first of those files with

mount /tmp/fstransform.mount.22858/.fstransform.loop.11631 /mnt

i.e. you just pass the file absolute path to mount as if it was a block device.

[aciavolino]

I removed those loop files in temp and tried again, same error.

[cosmos72]

See my updated comment above:

[...] this is a strange error because /dev/sda1 was mounted by fstranform itself, so it should not contain mount points for other partitions. What does ls -l efi show when run inside the filesystem you are converting? (you will need to mount it somewhere)

[aciavolino]

ls: cannot access efi: No such file or directory

[aciavolino]

Let me explain what I am trying to accomplish. Trying to convert from legacy to uefi, but I need to convert the xfs partition because currently I cannot shrink it.

[aciavolino]

This VM is a clone of a production machine so I pose no risk of losing real data.

[cosmos72]

Yes, converting xfs to ext[2,3,4] is supported.

It's just that efi entry that is giving problems, and the reason is not yet clear.
Can you post the output of ls -al MOUNT_POINT, where MOUNT_POINT is the directory where you mounted /dev/sda1 ?

[aciavolino]

Pardon my obtuse reaction, but its a boot partition. I am not really sure what that answer is.

[cosmos72]

Well, if you you deleted those files

  /tmp/fstransform.mount.22858/.fstransform.loop.11631 /tmp/fstransform.mount.22858/.fstransform.loop.11837

and so on, then your /dev/sda1 is still probably mounted under /tmp/fstransform.mount.22858

Then ls -al /tmp/fstransform.mount.22858/ should suffice.

[aciavolino]

[root@ctstageextweb2vm tmp]# ls -al /tmp/fstransform.mount.22858
ls: cannot access /tmp/fstransform.mount.22858: No such file or directory

[cosmos72]

Then I don't understand how you deleted those files

  /tmp/fstransform.mount.22858/.fstransform.loop.11631 /tmp/fstransform.mount.22858/.fstransform.loop.11837

etc.

[aciavolino]

The normal way, rm -rf. Gave me no issue.

[cosmos72]

hmm... rm -rf of what exactly?

[aciavolino]

The empty folders in /tmp
rm -rf fstransform.loop.*

[aciavolino]

Is this a permission thing?? I am running as root.

[cosmos72]

rm -rf does not complain about attempts to remove non-existing files, so your

cd /tmp
rm -rf fstransform.loop.*

probably did nothing:
those files were either inside the sub-directory /tmp/fstransform.mount.22858,
or inside the non-mounted partition /dev/sda1

Let's try to get back to a known setup: can you try to run the commands (still as root)

mount /dev/sda1 /mnt
ls -al /mnt

and post their output?

[aciavolino]

[root@ctstageextweb2vm tmp]# mount /dev/sda1 /mnt
[root@ctstageextweb2vm tmp]# ls -al /mnt
total 697480
dr-xr-xr-x. 5 root root 4096 Jul 20 08:22 .
dr-xr-xr-x. 18 root root 4096 Jul 19 16:09 ..
-rw-r--r--. 1 root root 153596 Dec 16 2021 config-3.10.0-1160.53.1.el7.x86_64
-rw-r--r-- 1 root root 153619 Mar 23 05:22 config-3.10.0-1160.62.1.el7.x86_64
-rw-r--r-- 1 root root 153619 Apr 27 16:59 config-3.10.0-1160.66.1.el7.x86_64
drwx------. 2 root root 6 Jul 19 17:00 efi
-rw-r--r-- 1 root root 1073741824 Jul 19 17:01 .fstransform.loop.11631
-rw-r--r-- 1 root root 1073741824 Jul 19 17:02 .fstransform.loop.11837
-rw-r--r-- 1 root root 1073741824 Jul 19 17:03 .fstransform.loop.12027
-rw-r--r-- 1 root root 1073741824 Jul 19 17:18 .fstransform.loop.12252
-rw-r--r-- 1 root root 1073741824 Jul 19 17:19 .fstransform.loop.12641
-rw-r--r-- 1 root root 1073741824 Jul 20 08:09 .fstransform.loop.22858
-rw-r--r-- 1 root root 1073741824 Jul 20 08:23 .fstransform.loop.23213
drwxr-xr-x 2 root root 6 Mar 8 04:19 grub
drwx------. 5 root root 97 Jul 19 16:08 grub2
-rw-------. 1 root root 61766223 Oct 5 2021 initramfs-0-rescue-5a10a9ccce434e41b39761cdfe88e684.img
-rw------- 1 root root 81382028 Apr 5 14:32 initramfs-0-rescue-67761415d8564d1dbe9698d476aad277.img
-rw------- 1 root root 79113091 Jul 19 16:04 initramfs-3.10.0-1160.53.1.el7.x86_64.img
-rw-------. 1 root root 13874596 Mar 8 04:29 initramfs-3.10.0-1160.53.1.el7.x86_64kdump.img
-rw------- 1 root root 78960004 Jul 19 16:06 initramfs-3.10.0-1160.62.1.el7.x86_64.img
-rw------- 1 root root 13873866 Apr 7 11:22 initramfs-3.10.0-1160.62.1.el7.x86_64kdump.img
-rw------- 1 root root 78967640 Jul 19 16:07 initramfs-3.10.0-1160.66.1.el7.x86_64.img
-rw------- 1 root root 13634999 Jul 19 16:10 initramfs-3.10.0-1160.66.1.el7.x86_64kdump.img
-rw-r--r--. 1 root root 320671 Dec 16 2021 symvers-3.10.0-1160.53.1.el7.x86_64.gz
-rw-r--r-- 1 root root 320668 Mar 23 05:23 symvers-3.10.0-1160.62.1.el7.x86_64.gz
-rw-r--r-- 1 root root 320651 Apr 27 17:00 symvers-3.10.0-1160.66.1.el7.x86_64.gz
-rw-------. 1 root root 3620596 Dec 16 2021 System.map-3.10.0-1160.53.1.el7.x86_64
-rw------- 1 root root 3621999 Mar 23 05:22 System.map-3.10.0-1160.62.1.el7.x86_64
-rw------- 1 root root 3621999 Apr 27 16:59 System.map-3.10.0-1160.66.1.el7.x86_64
-rwxr-xr-x. 1 root root 6769496 Oct 5 2021 vmlinuz-0-rescue-5a10a9ccce434e41b39761cdfe88e684
-rwxr-xr-x 1 root root 6781784 Apr 5 14:32 vmlinuz-0-rescue-67761415d8564d1dbe9698d476aad277
-rwxr-xr-x. 1 root root 6773592 Dec 16 2021 vmlinuz-3.10.0-1160.53.1.el7.x86_64
-rw-r--r--. 1 root root 172 Dec 16 2021 .vmlinuz-3.10.0-1160.53.1.el7.x86_64.hmac
-rwxr-xr-x 1 root root 6781784 Mar 23 05:22 vmlinuz-3.10.0-1160.62.1.el7.x86_64
-rw-r--r-- 1 root root 172 Mar 23 05:22 .vmlinuz-3.10.0-1160.62.1.el7.x86_64.hmac
-rwxr-xr-x 1 root root 6781784 Apr 27 16:59 vmlinuz-3.10.0-1160.66.1.el7.x86_64
-rw-r--r-- 1 root root 172 Apr 27 16:59 .vmlinuz-3.10.0-1160.66.1.el7.x86_64.hmac

[cosmos72]

Ok, now it starts to make sense :)

The efi subdirectory is visible and all those .fstransform.loop.* files are still there too.
Step 1: check what's inside the efi subdirectory:

ls -al /mnt/efi

and remove it if it's empty:

rmdir /mnt/efi

just because it was giving issues - otherwise there would be no need.

Step 2: check the content of all those .fstransform.loop.* files and copy back they contents, if any - I'll explain how.

Step 3: try again the conversion

[aciavolino]

[root@ctstageextweb2vm tmp]# ls -al /mnt/efi
total 4
drwx------. 2 root root 6 Jul 19 17:00 .
dr-xr-xr-x. 5 root root 4096 Jul 20 08:22 ..

[aciavolino]

[root@ctstageextweb2vm tmp]# rmdir /mnt/efi
rmdir: failed to remove ‘/mnt/efi’: Device or resource busy

[cosmos72]

So, that's the key issue: for some reason, that efi subdirectory cannot be removed even if it's empty.

The most common reason is having another partition mounted under it, see for example https://serverfault.com/questions/319727/cannot-delete-folder-with-rm-rf-error-device-or-resource-busy
But that's probably not your case, as you freshly mounted /dev/sda1.

For the same reason (freshly mounted partition) it's unlikely that some process is using it, in any case you can check with

lsof +D /mnt/efi

I'm starting to think about more unusual reasons - as a corrupted file system. Maybe dmesg | tail -40 contains some file system related error?

[aciavolino]

So lsof +D /mnt/efi had no output.

[aciavolino]

[root@ctstageextweb2vm tmp]# dmesg | tail -40
[ 17.639099] type=1305 audit(1658261327.668:3): audit_pid=863 old=0 auid=4294967295 ses=4294967295 res=1
[ 18.309558] hv_utils: VSS: userspace daemon ver. 129 connected
[ 26.955778] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 27.287154] Ebtables v2.0 registered
[ 27.464040] ip_set: protocol 7
[ 27.613378] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 27.614092] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 27.624832] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 27.626262] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 27.679515] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 27.681450] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 28.416777] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 28.731742] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 35.354612] hv_utils: KVP IC version 4.0
[ 56.466079] tun: Universal TUN/TAP device driver, 1.6
[ 56.466087] tun: (C) 1999-2004 Max Krasnyansky [email protected]
[ 56.468353] virbr0: port 1(virbr0-nic) entered blocking state
[ 56.468363] virbr0: port 1(virbr0-nic) entered disabled state
[ 56.468743] device virbr0-nic entered promiscuous mode
[ 56.788093] virbr0: port 1(virbr0-nic) entered blocking state
[ 56.788108] virbr0: port 1(virbr0-nic) entered listening state
[ 56.788572] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready
[ 56.960951] virbr0: port 1(virbr0-nic) entered disabled state
[ 61.551099] hv_balloon: Max. dynamic memory size: 8192 MB
[ 1118.287335] sd 3:0:0:0: [storvsc] Sense Key : Unit Attention [current]
[ 1118.287377] sd 3:0:0:0: [storvsc] Add. Sense: Changed operating definition
[ 1118.287417] sd 3:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa
[ 2457.560062] TCP: lp registered
[ 2693.166893] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 2873.055304] sd 2:0:0:0: [storvsc] Sense Key : Unit Attention [current]
[ 2873.055310] sd 2:0:0:0: [storvsc] Add. Sense: Changed operating definition
[ 2873.055313] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automa
[ 3121.747831] loop: module loaded
[ 3122.137536] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[ 3201.589097] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[ 3268.429598] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[ 3691.282475] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[ 4193.722359] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[57512.068241] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[58423.926153] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)

[cosmos72]

This is getting weirder. I am trying to guess the reason for this issue.
Maybe some daemon automatically mounts the EFI partition inside that efi subdirectory as soon as /dev/sda1 is mounted somewhere?
You can check with mount | grep /mnt

[aciavolino]

[root@ctstageextweb2vm tmp]# mount | grep /mnt
/dev/sda1 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)

[cosmos72]

This is expected, it's the /dev/sda1 mounted on /mnt that I instructed you to do.
But nothing mounted on /mnt/efi.

So there's still no evidence of why /mnt/efi cannot be removed, and I'm running out of ideas.

One thing I can tell is that it's not related to fstransform - it just happened to trigger the error when trying to remove that efi subdirectory.

I am quite confident it's not the solution, but out of pedantry, what does the following command print?

umount /mnt/efi

[aciavolino]

[root@ctstageextweb2vm tmp]# umount /mnt/efi
umount: /mnt/efi: not mounted
lol

[cosmos72]

As expected. I have run out of ideas and (almost) out of diagnostic tools at this point.

Maybe a file system check could help, but if there was some corruption it should have been shown in dmesg.
Anyway, the command to check an XFS file system is not the usual fsck. You can try:

umount /mnt
xfs_repair -v /dev/sda1

[aciavolino]

[root@ctstageextweb2vm tmp]# umount /mnt
[root@ctstageextweb2vm tmp]# xfs_repair -v /dev/sda1
xfs_repair: cannot open /dev/sda1: Device or resource busy

[aciavolino]

Man, its so odd. Just my luck I guess.

[cosmos72]

This is even more strange. Then where else is /dev/sda1 mounted? Try

mount | grep /dev/sda1

[aciavolino]

No output.

[cosmos72]

Ok, at least it's a clue.
That /dev/sda1 must be mounted somewhere else, and probably its subdirectory efi is used as mount point too.
But finding where depends on the machine: maybe the commands you are running are inside some container or chroot, and the master Linux file system (and installation) is not visible from there by default?

[aciavolino]

/etc/fstab
Created by anaconda on Tue Oct 5 15:21:45 2021

Accessible filesystems, by reference, are maintained under '/dev/disk'
See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

/dev/mapper/rhel_ctstageextweb2-root / xfs defaults 0 0
UUID=6ac64035-32a5-498d-bd77-e6fc6e913a89 /boot xfs defaults 0 0
/dev/mapper/rhel_ctstageextweb2-home /home xfs defaults 0 0
/dev/mapper/rhel_ctstageextweb2-swap swap swap defaults 0 0

Could this be a reason?

[cosmos72]

Yes, definitely.
Both fstransform and xfs_repair need exclusive access to the partition they use, and otherwise give errors - this explains most of the errors above.

One last thing to be clarified is what's mounted under /boot/efi. Hopefully

mount | grep /boot

will tell.

[aciavolino]

No output.

[aciavolino]

[root@ctstageextweb2vm mnt]# /sbin/fuser -m /mnt/efi
/mnt/efi: 11409c

Can you explain this?

[aciavolino]

Just want to say I really appreciate your input and help on this.

[cosmos72]

It means the process ID 11409 has /mnt/efi as current directory. The command ps -aux | grep 11409 should tell what is it.

[aciavolino]

[root@ctstageextweb2vm mnt]# ps -aux | grep 11409
root 11409 0.0 0.0 116732 3436 pts/0 S Jul19 0:00 -bash
root 25093 0.0 0.0 112820 984 pts/0 S+ 10:26 0:00 grep --color=auto 11409

[cosmos72]

That's your own shell, whose current directory is /mnt - nothing surprising.

I hoped some process would turn up using /mnt/efi, but unluckily your fuser -m /mnt/efi reported a slightly (but crucially) more generic result: process 11409 (your shell) is using the file system that contains /mnt/efi, which unluckily (because it does not help finding the root issue) is the same file system that contains /mnt.

At this point I can only recommend to move to a Linux support forum or chat, and ask for help about the error

[root@ctstageextweb2vm tmp]# rmdir /mnt/efi
rmdir: failed to remove ‘/mnt/efi’: Device or resource busy

and explaining that you cannot find anything mounted on /mnt/efi

[aciavolino]

Thanks for all the input.