Skip to main content

ec2

Glusterfs: Re-attach a crashed, inaccessible replicated server to the other peers

This morning started off okay, until I checked my email and found an alarm ringing about one of our file servers being offline. Luckily, we have a replicated setup, so no customers were affected, but it is a little worrisome not having the safety net of two replicated servers.

The server didn't technically crash, rather, it was a problem with the EC2 instance being unavailable likely due to some unforeseen problem in the Virginia datacenter. Regardless, the first step I took was to get the server accessible via SSH.


How to: Migrate an EC2 instances to a new availability zone

Recently an EC2 instance we were using in a production environment was scheduled to go down for maintenance. Normally this would be fine and we would let AWS just handle this. However, their scheduled maintenance window was set for 21:00 UTC, which equates to 17:00 EST. In other words, right at the end of the work day. Not cool!

In order to use a maintenance schedule of our choosing, we had to migrate our production instance into a different availability zone prior to Amazon's schedule. So, we setup a test instances and tested migrating. Our successful steps are below: