I recently was diagnosing a server where the disk drive kept filling up. What’s so odd about that? Well, running “du -sc * | sort -n” would show no directories larger than a couple gigs, and definitely nothing large enough to be exhausing the entire drive, yet df showed 80 gigs missing.
I finally tracked it down to a runaway squid process filling up the cache and log with entries. You see Linux won’t remove a file when it’s in use, even after you manually delete it. I was able to find the ghost file by running:
lsof -n | grep -i deleted
This command showed all open files that were deleted; IE taking up room on the disk without being visible in the file system. Simply restarting the Squid process was enough to free up the phantom space.