Reclaiming Disk Space on Root Volume by Shrinking Home in RHEL with XFS

Reclaiming Disk Space on Root Volume by Shrinking Home in RHEL with XFS
Photo by Denny Müller / Unsplash

Introduction

Lately, I found myself called on an incident where a critical security application was running on a Red Hat Enterprise Linux (RHEL) server. The disk had become fully saturated, causing the application’s unavailability. This was a cool, refreshing session of Linux commands for disk and partition manipulation.

The / root partition was full, but the /home partition had plenty of unused space. In this scenario, I needed to reduce the size of the /home partition and reallocate some of that space to the root partition. Here's how I did it.

Problem

When disk space runs out on the root partition, here’s what you might see:

$ df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root      199G  199G   20K 100% /
/dev/mapper/rhel-home      192G  3.1G  189G   2% /home

In this case, the root partition is full, but the /home partition has a lot of free space, which can be reallocated.

XFS File System Considerations

A key point when dealing with the XFS file system format (commonly used in RHEL) is that XFS can only be expanded, not shrunk, unlike the EXT4 file system format, which supports both expansion and shrinking. This means if you need to reduce an XFS partition, you will have to reformat it and therefore overwrite the data on it. So, always consider which file system is best suited for your needs when setting up a server. If shrinking partitions might be necessary in the future, EXT4 is likely a better choice.

Working with SSH: User Session Challenges

Another important consideration when resizing partitions via SSH is that if you’re logged in with a user whose home directory is mounted in /home, the system will prevent you from unmounting the partition.

Solutions:

  1. Hypervisor Access: Connect directly to the machine through your hypervisor (VMware, Hyper-V, etc.) as root and proceed with the unmount.
  2. SSH Root Login: Temporarily enable SSH root login to perform the operation, but be cautious, as enabling root access over SSH is not recommended in hardening guides.

Steps to Resize LVM with XFS

Since XFS doesn't support shrinking, we need to follow these steps:

  1. Backup the /home directory.
  2. Unmount the /home Logical Volume.
  3. Reduce the size of the /home logical volume.
  4. Reformat the /home logical volume.
  5. Copy Data Back to /home.
  6. Extend the /root logical volume.
  7. Grow the XFS File System on the / Root Partition.

Step 1: Backup the /home Directory

In my scenario, the system administrator from the IT production team added 20GB to the root partition disk. While they could have added more than 20GB, I didn't think it was necessary and preferred to reclaim the unused space from the /home partition instead. This approach allowed me to perform cleanup tasks once the application was available, providing immediate relief for the disk space issue and enabling me to complete the necessary backups without any disruptions.

Here’s how this was done:

  1. Increase the Disk Size: The administrator increased the virtual disk size through the hypervisor (e.g., adding 20GB).
  2. Rescan the Partition: After the disk size was increased, I rescanned the partition to recognize the new space without rebooting:
$ echo 1 | sudo tee /sys/class/block/sda/device/rescan
  1. Extend the Root Logical Volume: Then, I extended the root logical volume to utilize the added space:
$ lvextend -L +20G /dev/mapper/rhel-root
  1. Expand the XFS File System: Finally, I expanded the XFS file system to reflect the changes:
$ xfs_growfs /
  1. Create a Backup Directory:
$ mkdir /mnt/temp_home
  1. Use rsync to Back Up:
$ rsync -a /home/ /mnt/temp_home/

Now that I had more space available, I could perform necessary cleanups and backups of /home.

But you could also do this in other scenarios:

Option 2: Backup to a Third Partition

If you have another partition with sufficient space, you can back up /home to this partition.

  1. Create a Backup Directory:
$ mkdir /mnt/backup_home
  1. Mount the Partition:
  2. Use rsync to Back Up:
$ mount /dev/sda3 /mnt/backup_home
$ rsync -a /home/ /mnt/backup_home/
  1. Restore After Resizing:
$ rsync -a /mnt/backup_home/ /home/

Option 3: Backup via SCP

If you don’t have a spare partition or can’t temporarily increase the disk size, you can back up /home to another server using scp.

  1. Backup with scp:
$ scp -r /home/ user@remote-server:/path/to/backup/
  1. Restore After Resizing:
$ scp -r user@remote-server:/path/to/backup/ /home/

Step 2: Reduce the /home Logical Volume

Since XFS cannot be shrunk, we must first reduce the size of the logical volume and then reformat it.

First, ensure no processes are using /home:

$ lsof +D /home

After confirming nothing is using /home, proceed to unmount it:

$ umount /dev/mapper/rhel-home

Now, reduce the logical volume:

$ lvreduce -L 150G /dev/mapper/rhel-home

This reduces the /home logical volume by 150GB.

Step 3: Reformat the /home Logical Volume

Now, reformat the reduced logical volume with XFS:

$ mkfs.xfs /dev/mapper/rhel-home

Then remount it:

$ mount /dev/mapper/rhel-home /home

Step 4: Copy Data Back to /home

Once the /home logical volume has been resized, restore the files from the backup:

$ rsync -a /mnt/temp_home/ /home/

Then, remove the temporary directory:

$ rm -rf /mnt/temp_home

Step 5: Extend the Root Volume

Now that we’ve freed up 150GB from /home, we can add that space to the root volume:

$ lvextend -L +150G /dev/mapper/rhel-root

Afterward, grow the XFS file system on the root partition:

$ xfs_growfs /

Verifying the Changes

To ensure everything worked as expected, check the disk usage again:

$ df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root      349G  199G  150G  57% /
/dev/mapper/rhel-home       42G  3.1G   39G   8% /home

As shown, the root partition now has an additional 150GB, while the /home partition has been successfully reduced.

Conclusion

On a RHEL server, re-allocating disk space between logical volumes is a simple solution when the root partition runs out of space. With XFS's limitations on shrinking, the method involves moving the contents of /home, reducing and reformatting the logical volume, and then extending the root partition.

Always ensure to backup your data before performing disk operations to avoid data loss.