Zero Downtime Delayed Jobs

Here is a script I created that will do Zero Downtime Delayed Job restarts. This script will spin up a new set of delayed jobs workers and send a signal to the currently running workers to shut down after they are finished working on current work. The current script/delayed_job restart would only wait up to some timeout value and then would kill the job, potentially losing any work that it was working on. I store this in a file called delayed_job_restart.sh. Make sure you

chmod +x delayed_job_restart.sh

Feedback is welcome!

##############
CONFIGURATION
##############

# Set the rails environment.
RAILS_ENV=development

# This is the location of your rails application. (Rails.root)
APP_DIR=/path/to/app

# Whatever queues you want to handle.
QUEUES=queue1,queue2

# Number of delayed job instances.
NUMJOBS=6

########################
DONT EDIT BELOW HERE
UNLESS YOU UNDERSTAND IT
########################
# This is the file that will hold the location of
# the new round of delayed job pids.
NEW_PID_DIR_FILE=$APP_DIR/pids/delayed_job_pid_dir.new

# This is the directory where the new delayed job pids will live. We
# create a random directory.
NEW_PID_DIR=$APP_DIR/pids/delayed_job/`cat /dev/urandom | env LC_CTYPE=C tr -cd ‘a-f0-9’ | head -c 32`

# Make the directory and create the
# pid file.
mkdir -p $NEW_PID_DIR
touch $NEW_PID_DIR_FILE

# Put the pid dir into the pid file.
echo $NEW_PID_DIR > $NEW_PID_DIR_FILE

echo “Starting new delayed job processes…”
cd $APP_DIR
RAILS_ENV=$RAILS_ENV bundle exec script/delayed_job start –queues=$QUEUES –pid-dir=`cat $NEW_PID_DIR_FILE` -n $NUMJOBS
echo “Done.”

# This is the current pid dir.
PID_DIR_FILE=$APP_DIR/pids/delayed_job_pid_dir

# If the pid dir exists, let tell them to shut down
# when they are done.
if [ -f $PID_DIR_FILE ]; then
echo “Sending signal to stop old delayed job processes…”
PID_DIR=`cat $PID_DIR_FILE`
for f in $PID_DIR/*.pid
  do
    PID=`cat $f`
    echo ” TERM to $PID”
    kill -TERM $PID
  done
  echo “Done.”
  echo “Removing old PID directory.”
  rm -rf $PID_DIR
fi

# Since we have a set of new jobs running,
# lets move the pid dir to the “current” dir.
mv $NEW_PID_DIR_FILE $PID_DIR_FILE