Contact Sales Sitemap Customer Login

Automate Backups with Amazon S3 and s3sync

s3sync is a Ruby script which allows Amazon S3 account holders to inexpensively transfer large amounts of data to S3, which makes it an ideal solution for backup automation.

If you would like to give this solution a try, you will need an Amazon S3 account. Visit http://www.amazon.com/s3 to create an account and receive a Access Key ID and Secret Access Key.

Contents

Download and Install OpenSSL

  1. Enter mkdir ~/openssl; cd ~/openssl to create a directory to store your OpenSSL installation and switch to the directory
  2. Enter wget -S XXX to download the OpenSSL source code
  3. Enter tar xzvf openssl-0.9.8g.tar.gz; cd openssl-0.9.8g to unpack the source code and switch to the unpacked directory
  4. Enter ./config to begin compiling OpenSSL
  5. Enter make and then make install to complete the installation

Download and Install Ruby

  1. Enter mkdir ~/ruby; cd ~/ruby to create a directory to store your ruby installation and switch to the directory
  2. Enter wget -S ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p111.tar.gz to download Ruby 1.8.6 (stable)
  3. Enter tar xzvf ruby-1.8.6-p111.tar.gz; cd ruby-1.8.6-p111 to unpack the source code and switch to the unpacked directory
  4. Enter ./configure to begin compiling Ruby
  5. Enter make and then make install to complete the installation

Download and Install s3sync

  1. Enter cd ~ to switch to your home directory
  2. Enter wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz to download s3sync
  3. Enter tar xzvf s3sync.tar.gz; cd s3sync to unpack the source code and switch to the unpacked directory
  4. Enter mkdir certs; cd certs to create a directory for storing SSL certificates
  5. Enter wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar to download the certificates
  6. Enter sh ssl.certs.shar to install the certificates

Create s3sync Upload and Download Scripts

Important: The example scripts below make use of the --delete flag, which will delete any files in the sync target directory which do not exist in the sync source directory.

Files in the target directory bankers life and casualty company which differ from the source directory's files will be overwritten. We strongly recommend against using s3sync to synchronize system configuration files or users' home directories.

s3sync may be accessed directly from the command line with the following syntax:
ruby s3sync.rb -r --ssl --delete SOURCE_DIRECTORY TARGET_DIRECTORY

s3sync Upload Script

  1. Enter cd ~; mkdir s3sync.scripts; cd s3sync.scripts to switch to your home directory, create the s3sync.scripts directory, and switch to the new s3sync.scripts directory (this directory will be used to store your s3sync shell scripts)
  2. Create a file named upload.sh based upon the script below - you will need to make the following replacements:
    • LOCAL_PATH_TO_YOUR_SHELL_SCRIPT - The path to your upload.sh script (~/s3sync.scripts/upload.sh from prior instructions)
    • YOUR_S3_ACCESS_KEY - Your Amazon S3 Access Key
    • YOUR_S3_SECRET_KEY - Your Amazon S3 Secret Key
    • YOUR_S3SYNC_CERTS_PATH - The path to your certs directory (~/s3sync/cert from prior instructions)
    • LOCAL_PATH_TO_COPY - The local path which will be stored on S3
    • REMOTE_PATH_TO_STORE_SYNC - The path which will store files in your S3 filesystem
#!/bin/bash
# Copy local files to the Amazon S3 storage filesystem
cd LOCAL_PATH_TO_YOUR_SHELL_SCRIPT
export AWS_ACCESS_KEY_ID=YOUR_S3_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_S3_SECRET_KEY
export SSL_CERT_DIR=YOUR_S3SYNC_CERTS_PATH
# 
# Each directory which you would like to store on S3 must
# have an entry like this one:
# 
ruby s3sync.rb -r --ssl --delete LOCAL_PATH_TO_COPY mybucket:REMOTE_PATH_TO_STORE_SYNC
# 
# Example entry:
# ruby s3sync.rb -r --ssl --delete /backups mybucket:/backups

s3sync Download Script

  1. Enter cd ~/s3sync.scripts to switch to your s3sync.scripts directory
  2. Create a file named download.sh based upon the script below - you will need to make the following replacements:
    • LOCAL_PATH_TO_YOUR_SHELL_SCRIPT - The path to your upload.sh script (~/s3sync.scripts/upload.sh from prior instructions)
    • YOUR_S3_ACCESS_KEY - Your Amazon S3 Access Key
    • YOUR_S3_SECRET_KEY - Your Amazon S3 Secret Key
    • YOUR_S3SYNC_CERTS_PATH - The path to your certs directory (~/s3sync/cert from prior instructions)
    • REMOTE_PATH_TO_RETRIEVE - The path to stored files in your S3 filesystem
    • LOCAL_PATH_TO_UPDATE - The local path which will store retrieved files
#!/bin/bash
# Retrieve files from the Amazon S3 storage filesystem
cd LOCAL_PATH_TO_YOUR_SHELL_SCRIPT
export AWS_ACCESS_KEY_ID=YOUR_S3_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_S3_SECRET_KEY
export SSL_CERT_DIR=YOUR_S3SYNC_CERTS_PATH
# 
# Each directory which you would like to retrieve from S3
# must have an entry like this one:
# 
ruby s3sync.rb -r --ssl --delete mybucket:REMOTE_PATH_TO_RETRIEVE LOCAL_PATH_TO_UPDATE
# 
# Example entry:
# ruby s3sync.rb -r --ssl --delete mybucket:/backups ~/retrieved_from_s3

Related Links

Retrieved from "http://wiki.vpslink.com/index.php?title=Automate_Backups_with_Amazon_S3_and_s3sync&oldid=14512"
Recent Changes | RSS RSS Feed