Today I'm going to show you how to identify where all your available hard disk space has disappeared to in CentOS. You seem to have ample capacity, but your installation keeps reporting that it has run out of space even though you've cleared temp files etc.
I recently became aware of an issue where exceeding the iNodes limit in a CentOS installation will report that your disk is full. Apparently there is a limit of available iNodes so when you have a high usage machine, over time this limit will be reached unless you have some type of regular procedure to deal with this issue. To read in more detail what iNodes are, here is a Wikipedia link...
To find your iNode usage use the following syntax:
df -i
You should get a return something like this (I've put it in a table so it's easier to read and below the table is the actual output):
| Filesystem | Inodes | IUsed | IFree | IUse% | Mounted on |
|---|---|---|---|---|---|
| /dev/mapper/vg_mx-lv_root | 1150560 | 1150560 | 0 | 100% | / |
| tmpfs | 173284 | 1 | 173283 | 1% | /dev/shm |
| /dev/sda1 | 128016 | 56 | 127960 | 1% | /boot |
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg_mx-lv_root 1150560 1150560 0 100% / tmpfs 173284 1 173283 1% /dev/shm /dev/sda1 128016 56 127960 1% /boot
In our case the machine was used as an internet facing anti-spam MailScanner installation and the limit was reached because of the directory /var/spool/MailScanner/incoming/SpamAssassin-Temp having 1150560 iNodes.
So, how do you find the location of these iNodes? This is done by using the following command:
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
Thanks to the user Patrick, for contributing this post for the above syntax. Please be aware, that this can take quite a while to run, depending on your installation. This will list the directories in order of iNode volume.
In our case, as I outlined earlier, the directory was /var/spool/MailScanner/incoming/SpamAssassin-Temp and this is used by clamav and MailScanner to temporarily process incoming email.
So how to do you clear it? I decided to create a cron job that handles this directory at 4AM everyday. Open putty to connect to your machine and type in:
crontab -e
So if we want to run this task at say 4AM everyday, we would input the following:
0 4 * * * /sbin/service MailScanner stop graceful 5 4 * * * rm -rf /var/spool/MailScanner/incoming/SpamAssassin-Temp/* 10 4 * * * /sbin/service MailScanner start
The first line stops the MailScanner service gracefully at 4AM to ensure any current mail is processed before the service is stopped. The second line cleans up the SpamAssassin-Temp directory of all files at 4:05AM. The third and final line starts the MailScanner service again at 4.10AM.
This is a very quick and easy method to execute this task, and I'm sure there are a myriad of other ways to do this with various scripts, but this served our purpose perfectly. If you have any suggestions on how to improve this, please use the comment section below.
If you've found this useful, you may want to sign up to our newsletter where you'll receive notices on when we post new articles and helpful "how tos". Just fill out your details below and we'll do the rest...
