Reclaiming Disk Space on Root Volume by Shrinking Home in RHEL with XFS
Introduction
Lately, I found myself called on an incident where a critical security application was running on a Red Hat Enterprise Linux (RHEL) server. The disk had become fully saturated, causing the application’s unavailability. This was a cool, refreshing session of Linux commands for disk and partition manipulation.
The /
root partition was full, but the /home
partition had plenty of unused space. In this scenario, I needed to reduce the size of the /home
partition and reallocate some of that space to the root partition. Here's how I did it.
Problem
When disk space runs out on the root partition, here’s what you might see:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 199G 199G 20K 100% /
/dev/mapper/rhel-home 192G 3.1G 189G 2% /home
In this case, the root partition is full, but the /home
partition has a lot of free space, which can be reallocated.
XFS File System Considerations
A key point when dealing with the XFS file system format (commonly used in RHEL) is that XFS can only be expanded, not shrunk, unlike the EXT4 file system format, which supports both expansion and shrinking. This means if you need to reduce an XFS partition, you will have to reformat it and therefore overwrite the data on it. So, always consider which file system is best suited for your needs when setting up a server. If shrinking partitions might be necessary in the future, EXT4 is likely a better choice.
Working with SSH: User Session Challenges
Another important consideration when resizing partitions via SSH is that if you’re logged in with a user whose home directory is mounted in /home
, the system will prevent you from unmounting the partition.
Solutions:
- Hypervisor Access: Connect directly to the machine through your hypervisor (VMware, Hyper-V, etc.) as root and proceed with the unmount.
- SSH Root Login: Temporarily enable SSH root login to perform the operation, but be cautious, as enabling root access over SSH is not recommended in hardening guides.
Steps to Resize LVM with XFS
Since XFS doesn't support shrinking, we need to follow these steps:
- Backup the
/home
directory. - Unmount the
/home
Logical Volume. - Reduce the size of the
/home
logical volume. - Reformat the
/home
logical volume. - Copy Data Back to
/home
. - Extend the
/root
logical volume. - Grow the XFS File System on the
/
Root Partition.
Step 1: Backup the /home
Directory
In my scenario, the system administrator from the IT production team added 20GB to the root partition disk. While they could have added more than 20GB, I didn't think it was necessary and preferred to reclaim the unused space from the /home
partition instead. This approach allowed me to perform cleanup tasks once the application was available, providing immediate relief for the disk space issue and enabling me to complete the necessary backups without any disruptions.
Here’s how this was done:
- Increase the Disk Size: The administrator increased the virtual disk size through the hypervisor (e.g., adding 20GB).
- Rescan the Partition: After the disk size was increased, I rescanned the partition to recognize the new space without rebooting:
$ echo 1 | sudo tee /sys/class/block/sda/device/rescan
- Extend the Root Logical Volume: Then, I extended the root logical volume to utilize the added space:
$ lvextend -L +20G /dev/mapper/rhel-root
- Expand the XFS File System: Finally, I expanded the XFS file system to reflect the changes:
$ xfs_growfs /
- Create a Backup Directory:
$ mkdir /mnt/temp_home
- Use
rsync
to Back Up:
$ rsync -a /home/ /mnt/temp_home/
Now that I had more space available, I could perform necessary cleanups and backups of /home
.
But you could also do this in other scenarios:
Option 2: Backup to a Third Partition
If you have another partition with sufficient space, you can back up /home
to this partition.
- Create a Backup Directory:
$ mkdir /mnt/backup_home
- Mount the Partition:
- Use
rsync
to Back Up:
$ mount /dev/sda3 /mnt/backup_home
$ rsync -a /home/ /mnt/backup_home/
- Restore After Resizing:
$ rsync -a /mnt/backup_home/ /home/
Option 3: Backup via SCP
If you don’t have a spare partition or can’t temporarily increase the disk size, you can back up /home
to another server using scp
.
- Backup with
scp
:
$ scp -r /home/ user@remote-server:/path/to/backup/
- Restore After Resizing:
$ scp -r user@remote-server:/path/to/backup/ /home/
Step 2: Reduce the /home
Logical Volume
Since XFS cannot be shrunk, we must first reduce the size of the logical volume and then reformat it.
First, ensure no processes are using /home
:
$ lsof +D /home
After confirming nothing is using /home
, proceed to unmount it:
$ umount /dev/mapper/rhel-home
Now, reduce the logical volume:
$ lvreduce -L 150G /dev/mapper/rhel-home
This reduces the /home
logical volume by 150GB.
Step 3: Reformat the /home
Logical Volume
Now, reformat the reduced logical volume with XFS:
$ mkfs.xfs /dev/mapper/rhel-home
Then remount it:
$ mount /dev/mapper/rhel-home /home
Step 4: Copy Data Back to /home
Once the /home
logical volume has been resized, restore the files from the backup:
$ rsync -a /mnt/temp_home/ /home/
Then, remove the temporary directory:
$ rm -rf /mnt/temp_home
Step 5: Extend the Root Volume
Now that we’ve freed up 150GB from /home
, we can add that space to the root volume:
$ lvextend -L +150G /dev/mapper/rhel-root
Afterward, grow the XFS file system on the root partition:
$ xfs_growfs /
Verifying the Changes
To ensure everything worked as expected, check the disk usage again:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rhel-root 349G 199G 150G 57% /
/dev/mapper/rhel-home 42G 3.1G 39G 8% /home
As shown, the root partition now has an additional 150GB, while the /home
partition has been successfully reduced.
Conclusion
On a RHEL server, re-allocating disk space between logical volumes is a simple solution when the root partition runs out of space. With XFS's limitations on shrinking, the method involves moving the contents of /home
, reducing and reformatting the logical volume, and then extending the root partition.
Always ensure to backup your data before performing disk operations to avoid data loss.