dpkg No space left on device

Just being trying to update a few packages and ran in to the following:

*dpkg: unrecoverable fatal error, aborting:
failed to write status record about `package-name-here to `/var/lib/dpkg/status': No space left on device 

Webmin also reported:

Failed to open /etc/webmin/webmin/oscache for writing : No such file or directory

A quick disk space check showed nothing unusual:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              75G   16G   58G  22% /
tmpfs                1021M     0 1021M   0% /lib/init/rw
udev                  989M   72K  989M   1% /dev
tmpfs                1021M     0 1021M   0% /dev/shm

BUT a quick inode check showed it was at 100% for the primary disk:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            2490368 2490368       0  100% /
tmpfs                  63534       4   63530    1% /lib/init/rw
udev                   63534     135   63399    1% /dev
tmpfs                  63534       3   63531    1% /dev/shm

So, lets hunt the inodes:

cd /
for i in `ls -1A`; do echo "`find $i | sort -u | wc -l` $i"; done | sort -rn | head -20

This resulted in the following output:

92 bin
70979 usr
66330 home
3870 lib
35 root
25903 proc
2305414 var

So we now recurse in to the "var" directory as it looks to be the largest one:

cd /var
for i in `ls -1A`; do echo "`find $i | sort -u | wc -l` $i"; done | sort -rn | head -20

2296893 spool
5616 lib
2745 cache
225 log
27 run
19 backups
10 www

Damn that hack leaving postfix vulnerable to being exploited:

cd /var/spool/postfix
for i in `ls -1A`; do echo "`find $i | sort -u | wc -l` $i"; done | sort -rn | head -20

1105135 maildrop
592722 defer
592577 deferred
6403 incoming
31 bounce
25 private

And NOT to my surprise we have sqillions of waiting/bounced mail. Time to delete having disabled postfix:

ionice -c 3 find /var/spool/postfix/maildrop -type f -delete
ionice -c 3 find /var/spool/postfix/defer -type f -delete
ionice -c 3 find /var/spool/postfix/deferred -type f -delete
ionice -c 3 find /var/spool/postfix/incoming -type f -delete
ionice -c 3 find /var/spool/postfix/bounce -type f -delete
ionice -c 3 find /var/spool/postfix/active -type f -delete

Also timed deleting the 1.1 million SPAM related email files and it still took:

# time ionice -c 3 find /var/spool/postfix

real    180m9.333s
user    0m7.550s
sys     1m38.910s

The after effects of clearing several million files shows the following when running df -i:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            2490368  321549 2168819   13% /
tmpfs                  63534       4   63530    1% /lib/init/rw
udev                   63534     135   63399    1% /dev
tmpfs                  63534       3   63531    1% /dev/shm

Check this for more inode info: http://www.nzlinux.com/2010/06/inode-problems-and-full-disks/