Skip to main content


GlusterFS Network Issues

Not sure what happened with GlusterFS tonight. Around 19:07 EST, the AWS alarm sounded. After rebooting the GlusterFS servers and the web servers, nothing was bringing the sites back online. There was a terrible lag when accessing the GlusterFS file systems and there weren't any glaring errors in the log files.

Glusterfs: Re-attach a crashed, inaccessible replicated server to the other peers

This morning started off okay, until I checked my email and found an alarm ringing about one of our file servers being offline. Luckily, we have a replicated setup, so no customers were affected, but it is a little worrisome not having the safety net of two replicated servers.

The server didn't technically crash, rather, it was a problem with the EC2 instance being unavailable likely due to some unforeseen problem in the Virginia datacenter. Regardless, the first step I took was to get the server accessible via SSH.