lots of changes for version 4 - ready for final testing then tag it

This commit is contained in:
Damon Timm 2010-03-05 20:10:31 -05:00
parent a843a4f3a5
commit 4e9b44ddc1
3 changed files with 109 additions and 124 deletions

2
AUTHORS Normal file
View file

@ -0,0 +1,2 @@
Damon Timm <damontimm@gmail.com> <http://blog.damontimm.com>
Mario Santagiuliana <mario@marionline.it> <http://www.marionline.it>

89
README
View file

@ -1,53 +1,72 @@
**NOTE: currently, the script is undergoing some changes! You should use version 3 (0.3) for the time being until we have all the bugs/kinks sorted out. ABOUT THIS SCRIPT
=================
More information about this script avaiable at: http://damontimm.com/code/dt-s3-backup This bash script was designed to automate and simplify the remote backup
process of duplicity on Amazon S3. After your script is configured, you can
easily backup, restore, verify and clean (either via cron or manually) your
data without having to remember lots of different command options and
passphrases.
==TO DO== Most importantly, you can easily backup the script and your gpg key in a
convienent passphrase-encrypted file. This comes in in handy if/when your
machine ever does go belly up.
* Add a README file to the "--backup-this-script" option because, if you are like me, when it is working for so long I forget what to do when it comes time to restore More information about this script avaiable at:
* Add --restore-dir option http://damontimm.com/code/dt-s3-backup
* --restore-file restores to the *original* location if not alternate location is specified ... may make more sense to have it restore the current working directory first.
* add option to be able to restore a file from a specific time period (not just the latest version)
=== USAGE EXAMPLE === Latest version of the code is available at:
You must configure the script before using it you can config it with your favorites test editor(read comments in script). http://github.com/thornomad/dt-s3-backup
You can put the script under bin directory of your user. If you have configured correctly your shell you will call dt-s3-backup.sh whenever you want.
Then make the script executable with "chmod +x".
==Common usage== BEFORE YOU START
================
= View help This is a bash script that requires user configuration. Instructions are in
$ dt-s3-backup.sh the file itself and should be self-explanatory. Be sure to replace all the
*foobar* values with your real ones. Almost every value needs to be
configured.
= Run an incremental backup of configured directory After the script is configured, you can move it to /usr/local/sbin/ or wherever
you please and run it from there. A cron job is always nice. Be sure to make
the script executable (chmod +x) before you hit the gas.
COMMON USAGE EXAMPLES
=====================
* View help:
$ dt-s3-backup.sh
* Run an incremental backup:
$ dt-s3-backup.sh --backup $ dt-s3-backup.sh --backup
= Restore your backup * Restore your entire backup:
To restore the backup in the current work directory just run: $ dt-s3-backup.sh --restore
$ dt-s3-backup.sh --restore ./ You will be prompted for a restore directory
You can provide another directory to restore, for example: I want to restore my backup in "my_restore" folder (not created yet or existing): $ dt-s3-backup.sh --restore /home/user/restore-folder
$ dt-s3-backup.sh --restore my_restore You can also provide a restore folder on the command line.
If you don't provide a path you will prompt to answare some question to restore your backup * Restore a specific file in the backup:
$ dt-s3-backup.sh --restore-file
You will be prompted for a file to restore to the current directory
==Special usage== $ dt-s3-backup.sh --restore-file img/mom.jpg
Restores the file img/mom.jpg to the current directory
= List files beckend up in the remote archive $ dt-s3-backup.sh --restore-file img/mom.jpg /home/user/i-love-mom.jpg
Restores the file img/mom.jpg to /home/user/i-love-mom.jpg
* List files in the remote archive
$ dt-s3-backup.sh --list-current-files $ dt-s3-backup.sh --list-current-files
= Restore a specific file * Verify the backup
I have deleted my favourites picture file 'mygirlfriend.jpg' present in my "Pictures" home directory, I backup it with dt-s3-backup and I can restore it with: $ dt-s3-backup.sh --verify
$ dt-s3-backup.sh --restore-file Pictures/mygirlfriend.jpg
Now mygirlfriend.jpg is under "Pictures" directory.
But if I want to change the name when I restore it? I just need to provide a new name:
$ dt-s3-backup.sh --restore-file Pictures/mygirlfriend.jpg Pictures/myExgirlfriend.jpg
In my Pictures directory I have myExgirlfriend.jpg. NEXT VERSION WISH LIST
Attention: if "Pictures" directory doesn't exist you have got an error. You must provide an existing directory. ======================
= Verifies the backup
$ dt-s3-backup.sh --verify
Then you can view the verification opening the log file (that you have previously configure).
* --restore-dir option
* restore files from a specific time period
Thanks to Mario Santagiuliana for his help.

View file

@ -15,49 +15,22 @@
# #
# You should have received a copy of the GNU General Public License along with # You should have received a copy of the GNU General Public License along with
# this program. If not, see <http://www.gnu.org/licenses/>. # this program. If not, see <http://www.gnu.org/licenses/>.
# ---------------------------------------------------------------------------- #
# AUTHORS:
# Damon Timm <damontimm@gmail.com> <http://blog.damontimm.com>
# Mario Santagiuliana <mario@marionline.it> <http://www.marionline.it>
# VERSION 4 NOTE (02/12/2010):
# **Code is still being tested - if you want the last stable version, please
# download verion 0.3!**
# ABOUT THIS SCRIPT:
# #
# This bash script was designed to automate and simplify the remote backup # MORE ABOUT THIS SCRIPT AVAILABLE IN THE README AND AT:
# process using duplicity and Amazon S3. Hopefully, after the script is
# configured, you can easily backup, restore, verify and clean without having
# to remember lots of different command options.
#
# Furthermore, you can even automate the process of saving your script and the
# gpg key for your backups in a single password-protected file -- this way, you
# know you have everything you need for a restore, in case your machine goes
# down.
#
# You can run the script from cron with no command-line options (all options
# set in the script itself); however, you can also run it outside of the cron
# with some variables for more control.
# MORE INFORMATION:
# #
# http://damontimm.com/code/dt-s3-backup # http://damontimm.com/code/dt-s3-backup
#
# ---------------------------------------------------------------------------- # # ---------------------------------------------------------------------------- #
# AMAZON S3 INFORMATION # AMAZON S3 INFORMATION
export AWS_ACCESS_KEY_ID="foobar_aws_key_id" export AWS_ACCESS_KEY_ID="foobar_aws_key_id"
export AWS_SECRET_ACCESS_KEY="foobar_aws_access_key" export AWS_SECRET_ACCESS_KEY="foobar_aws_access_key"
# GPG PASSPHRASE & GPG KEY (Automatic/Cron Usage)
# If you aren't running this from a cron, comment this line out # If you aren't running this from a cron, comment this line out
# and duplicity should prompt you for your password. # and duplicity should prompt you for your password.
export PASSPHRASE="foobar_gpg_passphrase" export PASSPHRASE="foobar_gpg_passphrase"
# Specify which GPG key you would like to use (even if you have only one).
GPG_KEY="foobar_gpg_key" GPG_KEY="foobar_gpg_key"
# The ROOT of your backup (where you want the backup to start); # The ROOT of your backup (where you want the backup to start);
@ -71,15 +44,14 @@ ROOT="/home/"
# for you). If you don't want to use Amazon S3, you can backup # for you). If you don't want to use Amazon S3, you can backup
# to a file or any of duplicity's supported outputs. # to a file or any of duplicity's supported outputs.
# #
# NOTE: You do need to keep the "s3+http://<your location>/" format; # NOTE: You do need to keep the "s3+http://<your location>/" format
# even though duplicity supports "s3://<your location>/" this script # even though duplicity supports "s3://<your location>/".
# needs to read the former.
DEST="file:///home/foobar_user_name/new-backup-test/"
#DEST="s3+http://backup-bucket/backup-folder/" #DEST="s3+http://backup-bucket/backup-folder/"
DEST="file:///home/foobar_user_name/new-backup-test/"
# INCLUDE LIST OF DIRECTORIES # INCLUDE LIST OF DIRECTORIES
# Here is a list of directories to include; if you want to include # Here is a list of directories to include; if you want to include
# everything that is in root, you could leave this list empty, I think. # everything that is in root, you could leave this list empty (I think).
#INCLIST=( "/home/*/Documents" \ #INCLIST=( "/home/*/Documents" \
# "/home/*/Projects" \ # "/home/*/Projects" \
# "/home/*/logs" \ # "/home/*/logs" \
@ -98,8 +70,8 @@ EXCLIST=( "/home/*/Trash" \
# STATIC BACKUP OPTIONS # STATIC BACKUP OPTIONS
# Here you can define the static backup options that you want to run with # Here you can define the static backup options that you want to run with
# duplicity. I use both the full-if-older-than option plus the # duplicity. I use both the `--full-if-older-than` option plus the
# --s3-use-new-style option (for European buckets). Be sure to separate your # `--s3-use-new-style` option (for European buckets). Be sure to separate your
# options with appropriate spacing. # options with appropriate spacing.
STATIC_OPTIONS="--full-if-older-than 14D --s3-use-new-style" STATIC_OPTIONS="--full-if-older-than 14D --s3-use-new-style"
@ -109,16 +81,12 @@ STATIC_OPTIONS="--full-if-older-than 14D --s3-use-new-style"
# backups leave room for problems in the chain, so doing a "full" # backups leave room for problems in the chain, so doing a "full"
# backup every so often isn't not a bad idea. # backup every so often isn't not a bad idea.
# #
# I set the default to do a full backup every 14 days and to remove all # You can either remove older than a specific time period:
# all files over 31 days old. This should leave me at least two full
# backups available at any time, as well as a month's worth of incremental
# data.
#CLEAN_UP_TYPE="remove-older-than" #CLEAN_UP_TYPE="remove-older-than"
#CLEAN_UP_VARIABLE="31D" #CLEAN_UP_VARIABLE="31D"
# If you would rather keep a certain (n) number of full backups (rather # Or, If you would rather keep a certain (n) number of full backups (rather
# than removing the files based on their age), uncomment the following # than removing the files based on their age), you can use what I use:
# two lines and select the number of full backups you want to keep.
CLEAN_UP_TYPE="remove-all-but-n-full" CLEAN_UP_TYPE="remove-all-but-n-full"
CLEAN_UP_VARIABLE="2" CLEAN_UP_VARIABLE="2"
@ -127,7 +95,6 @@ CLEAN_UP_VARIABLE="2"
# I run this script as root, but save the log files under my user name -- # I run this script as root, but save the log files under my user name --
# just makes it easier for me to read them and delete them as needed. # just makes it easier for me to read them and delete them as needed.
# LOGDIR="/dev/null"
LOGDIR="/home/foobar_user_name/logs/test2/" LOGDIR="/home/foobar_user_name/logs/test2/"
LOG_FILE="duplicity-`date +%Y-%m-%d-%M`.txt" LOG_FILE="duplicity-`date +%Y-%m-%d-%M`.txt"
LOG_FILE_OWNER="foobar_user_name:foobar_user_name" LOG_FILE_OWNER="foobar_user_name:foobar_user_name"
@ -148,7 +115,7 @@ size information unavailable."
NO_S3CMD_CFG="WARNING: s3cmd is not configured, run 's3cmd --configure' \ NO_S3CMD_CFG="WARNING: s3cmd is not configured, run 's3cmd --configure' \
in order to retrieve remote file size information. Remote file \ in order to retrieve remote file size information. Remote file \
size information unavailable." size information unavailable."
README_TXT="This is the README file for the backup-this-script.\n\nIt needs work still." README_TXT="In case you've long forgotten, this is a backup script that you used to backup some files (most likely remotely at Amazon S3). In order to restore these files, you first need to import your GPG private key (if you haven't already). The key is in this directory and the following command should do the trick:\n\ngpg --allow-secret-key-import --import s3-secret.key.txt\n\nAfter your key as been succesfully imported, you should be able to restore your files.\n\nGood luck!"
CONFIG_VAR_MSG="Oops!! ${0} was unable to run!\nWe are missing one or more important variables at the top of the script.\nCheck your configuration because it appears that something has not been set yet." CONFIG_VAR_MSG="Oops!! ${0} was unable to run!\nWe are missing one or more important variables at the top of the script.\nCheck your configuration because it appears that something has not been set yet."
if [ ! -x "$DUPLICITY" ]; then if [ ! -x "$DUPLICITY" ]; then
@ -290,9 +257,10 @@ backup_this_script()
echo "Encrypting tarball, choose a password you'll remember..." echo "Encrypting tarball, choose a password you'll remember..."
tar c ${TMPDIR} | gpg -aco ${TMPFILENAME} tar c ${TMPDIR} | gpg -aco ${TMPFILENAME}
rm -Rf ${TMPDIR} rm -Rf ${TMPDIR}
echo echo -e "\nIMPORTANT!!"
echo ">> To restore these files, run the following (remember your password!):" echo ">> To restore these files, run the following (remember your password):"
echo "gpg -d ${TMPFILENAME} | tar x" echo "gpg -d ${TMPFILENAME} | tar x"
echo -e "\nYou may want to write the above down and save it with the file."
} }
check_variables () check_variables ()
@ -328,16 +296,17 @@ elif [ "$1" = "--verify" ]; then
DEST=${OLDROOT} DEST=${OLDROOT}
OPTION="verify" OPTION="verify"
echo "-------[ Verifying Source & Destination ]-------" >> ${LOGFILE} echo -e "-------[ Verifying Source & Destination ]-------\n" >> ${LOGFILE}
include_exclude include_exclude
duplicity_backup duplicity_backup
echo >> ${LOGFILE}
#restore previous condition
OLDROOT=${ROOT} OLDROOT=${ROOT}
ROOT=${DEST} ROOT=${DEST}
DEST=${OLDROOT} DEST=${OLDROOT}
get_file_sizes get_file_sizes
echo -e "Verify complete. Check the log file for resultst:\n${LOGFILE}"
echo -e "Verify complete. Check the log file for results:\n>> ${LOGFILE}"
elif [ "$1" = "--restore" ]; then elif [ "$1" = "--restore" ]; then
check_variables check_variables
@ -372,37 +341,32 @@ elif [ "$1" = "--restore-file" ]; then
OPTION= OPTION=
if [[ ! "$2" ]]; then if [[ ! "$2" ]]; then
echo "Please provide file to restore (eg. Mail/article):" echo "Which file do you want to restore (eg, mail/letter.txt):"
read -e FILE_TO_RESTORE read -e FILE_TO_RESTORE
FILE_TO_RESTORE=$FILE_TO_RESTORE FILE_TO_RESTORE=$FILE_TO_RESTORE
echo "Would you like to provide new filename? [No/...path/new_name]:" echo
read -e DEST_PATH
if [ "$DEST_PATH" = "No" ]; then
DEST=$FILE_TO_RESTORE
else
DEST=$DEST_PATH
fi
echo ">> You will restore your $FILE_TO_RESTORE to ${DEST} from ${ROOT}."
echo ">> You can override this question by executing '--restore-file [file] [[path/new_name]]' next time"
echo "Are you sure you want to do that ('yes' to continue)?"
read ANSWER
if [ "$ANSWER" != "yes" ]; then
echo "You said << ${ANSWER} >> so I am exiting now."
echo -e "-------- END --------\n" >> ${LOGFILE}
exit 1
fi
echo "Restoring now ..."
else else
FILE_TO_RESTORE=$2 FILE_TO_RESTORE=$2
if [[ "$3" ]]; then
DEST=$3
else
# TODO - this restores it to the original location, I think would be
# better if we restored it to the current working directory if no file
# name is given
DEST=$FILE_TO_RESTORE
fi
fi fi
if [[ "$3" ]]; then
DEST=$3
else
DEST=$(basename $FILE_TO_RESTORE)
fi
echo -e "YOU ARE ABOUT TO..."
echo -e ">> RESTORE: $FILE_TO_RESTORE"
echo -e ">> TO: ${DEST}"
echo -e "\nAre you sure you want to do that ('yes' to continue)?"
read ANSWER
if [ "$ANSWER" != "yes" ]; then
echo "You said << ${ANSWER} >> so I am exiting now."
echo -e "-------- END --------\n" >> ${LOGFILE}
exit 1
fi
echo "Restoring now ..."
#use INCLUDE variable without create another one #use INCLUDE variable without create another one
INCLUDE="--file-to-restore ${FILE_TO_RESTORE}" INCLUDE="--file-to-restore ${FILE_TO_RESTORE}"
duplicity_backup duplicity_backup
@ -431,20 +395,20 @@ else
--backup: runs an incremental backup --backup: runs an incremental backup
--full: forces a full backup --full: forces a full backup
--verify: verifies the backup (no cleanup is run) --verify: verifies the backup
--restore [path]: restores the backup to specified path --restore [path]: restores the entire backup
--restore-file [file] [path/filename]: restore a specific file, optional you can provide a destination name --restore-file [file] [destination/filename]: restore a specific file
--list-current-files: lists the files currently backed up in the archive. --list-current-files: lists the files currently backed up in the archive
--backup-script: let's you backup the script and secret key to the current working directory --backup-script: automatically backup the script and secret key to the current working directory
CURRENT VARIABLES: CURRENT SCRIPT VARIABLES:
========================
DEST (backup destination) = ${DEST} DEST (backup destination) = ${DEST}
INCLIST (directories that will be backed up) = ${INCLIST[@]:0} INCLIST (directories included) = ${INCLIST[@]:0}
EXCLIST (directory that will not be backup) = ${EXCLIST[@]:0} EXCLIST (directories excluded) = ${EXCLIST[@]:0}
ROOT (root directory) = ${ROOT} ROOT (root directory of backup) = ${ROOT}
" "
fi fi
echo -e "-------- END DT-S3-BACKUP SCRIPT --------\n" >> ${LOGFILE} echo -e "-------- END DT-S3-BACKUP SCRIPT --------\n" >> ${LOGFILE}