Our hosting company allocated an additional 100GB to our VMWare Ubuntu server. In order to use this newly allocated space we needed to follow the steps below:
Today we noticed a weird error where our CMS software was unable to write files to the server. Upon investigating, it appears the /tmp directory filled up the hard drive, which prevented file writes to the server.
We rebooted the server and it took 20-30 minutes to reboot. The AWS boot logs were saying:
"A start job is running for Create Volatile files and directories"
As it turns out, the /tmp directory is only cleared on reboot. Since there were so many temporary files on the disk, it took that long for the server to delete the files before coming back online.
We are running an Ubuntu 12.04 server with the Aegir hosting system and recently upgraded the server software with apt-get. The next day we noticed that the cron queue wasn't running, so we began to investigate. It turns out that the Drush package for Ubuntu had recently been upgraded to Drush 5.9.1, which is incompatible with Aegir 1.x version.
Today I learned a lesson the hard way: never stop networking when your only access to a server is via ssh.